Sep 16 01:16:07.849695 ip-10-0-137-55 systemd[1]: Starting Kubernetes Kubelet... Sep 16 01:16:08.290773 ip-10-0-137-55 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 01:16:08.290773 ip-10-0-137-55 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 01:16:08.290773 ip-10-0-137-55 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 01:16:08.290773 ip-10-0-137-55 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 01:16:08.290773 ip-10-0-137-55 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 01:16:08.292605 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.292529 2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 01:16:08.295811 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295797 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 01:16:08.295811 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295812 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295815 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295819 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295821 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295824 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295826 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295829 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295832 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295835 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295837 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295839 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295842 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295848 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295853 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295856 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295858 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295861 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295864 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295866 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295869 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 01:16:08.295873 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295871 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295874 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295877 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295880 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295883 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295886 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295889 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295891 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295895 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295899 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295902 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295905 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295907 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295910 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295912 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295915 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295918 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295920 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295923 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295925 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 01:16:08.296344 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295928 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295931 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295934 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295938 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295940 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295942 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295946 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295948 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295951 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295953 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295956 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295958 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295961 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295963 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295967 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295969 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295972 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295975 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295978 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295981 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 01:16:08.296934 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295983 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295986 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295988 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295991 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295993 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295996 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.295998 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296001 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296003 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296007 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296010 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296012 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296014 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296017 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296019 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296021 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296024 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296027 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296029 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296031 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 01:16:08.297542 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296034 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296036 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296038 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296041 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296043 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296047 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296051 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296055 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296058 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296060 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296063 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296066 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296083 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296087 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296090 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296093 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296096 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296098 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296769 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 01:16:08.298291 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296776 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296779 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296781 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296785 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296787 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296790 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296793 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296795 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296798 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296800 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296803 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296806 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296808 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296810 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296813 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296815 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296818 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296820 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296823 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296826 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 01:16:08.298770 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296829 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296832 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296835 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296837 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296840 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296843 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296845 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296848 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296850 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296853 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296855 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296858 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296860 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296863 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296865 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296867 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296870 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296872 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296875 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296877 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 01:16:08.299272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296880 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296882 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296885 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296887 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296890 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296893 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296895 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296898 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296900 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296902 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296904 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296908 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296911 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296914 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296916 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296918 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296921 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296924 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296926 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296929 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 01:16:08.299796 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296931 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296934 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296936 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296939 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296942 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296944 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296947 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296949 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296952 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296957 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296960 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296964 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296967 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296970 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296973 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296976 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296980 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296982 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296985 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 01:16:08.300290 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296987 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296990 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296992 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296994 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.296998 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297001 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297003 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297006 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297008 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297011 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297014 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297018 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297020 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297023 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297026 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297028 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297031 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297033 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.297035 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 01:16:08.300775 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299166 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299177 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299180 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299183 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299186 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299189 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299192 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299196 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299200 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299203 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299206 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299209 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299211 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299214 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299216 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299219 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299222 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299224 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299227 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 01:16:08.301259 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299229 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299232 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299234 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299237 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299239 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299242 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299245 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299247 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299249 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299252 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299254 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299257 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299260 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299263 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299266 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299269 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299272 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299274 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299277 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299280 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 01:16:08.301739 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299282 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299285 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299290 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299293 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299297 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299300 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299302 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299305 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299308 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299310 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299312 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299316 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299320 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299323 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299326 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299329 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299331 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299334 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299336 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 01:16:08.302236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299339 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299341 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299344 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299346 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299349 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299351 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299354 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299357 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299360 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299363 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299365 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299368 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299370 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299373 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299375 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299378 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299380 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299383 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299385 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299389 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 01:16:08.302737 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299391 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299394 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299396 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299399 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299401 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299404 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299406 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299409 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299424 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299427 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299429 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299446 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299449 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299452 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299455 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299458 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299461 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299464 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299467 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299469 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 01:16:08.303207 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.299472 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299541 2558 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299575 2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299582 2558 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299586 2558 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299590 2558 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299593 2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299597 2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299601 2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299604 2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299606 2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299610 2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299613 2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299616 2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299618 2558 flags.go:64] FLAG: --cgroup-root="" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299621 2558 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299624 2558 flags.go:64] FLAG: --client-ca-file="" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299626 2558 flags.go:64] FLAG: --cloud-provider="external" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299630 2558 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299635 2558 flags.go:64] FLAG: --cluster-domain="" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299638 2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299641 2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299644 2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299646 2558 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 01:16:08.303725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299650 2558 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299653 2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299656 2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299659 2558 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299661 2558 flags.go:64] FLAG: --contention-profiling="false" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299664 2558 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299667 2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299671 2558 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299673 2558 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299677 2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299680 2558 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299683 2558 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299686 2558 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299689 2558 flags.go:64] FLAG: --enable-server="true" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299691 2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299696 2558 flags.go:64] FLAG: --event-burst="100" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299698 2558 flags.go:64] FLAG: --event-qps="50" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299701 2558 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299704 2558 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299707 2558 flags.go:64] FLAG: --eviction-hard="" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299711 2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299713 2558 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299716 2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299719 2558 flags.go:64] FLAG: --eviction-soft="" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299722 2558 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 01:16:08.304344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299724 2558 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299727 2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299731 2558 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299733 2558 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299736 2558 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299739 2558 flags.go:64] FLAG: --feature-gates="" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299742 2558 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299745 2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299748 2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299751 2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299754 2558 flags.go:64] FLAG: --healthz-port="10248" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299756 2558 flags.go:64] FLAG: --help="false" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299759 2558 flags.go:64] FLAG: --hostname-override="ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299763 2558 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299766 2558 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299769 2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299772 2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299775 2558 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299778 2558 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299781 2558 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299783 2558 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299786 2558 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299789 2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299791 2558 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 01:16:08.304979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299794 2558 flags.go:64] FLAG: --kube-reserved="" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299797 2558 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299800 2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299803 2558 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299806 2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299809 2558 flags.go:64] FLAG: --lock-file="" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299812 2558 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299815 2558 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299818 2558 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299823 2558 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299826 2558 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299829 2558 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299832 2558 flags.go:64] FLAG: --logging-format="text" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299834 2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299838 2558 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299840 2558 flags.go:64] FLAG: --manifest-url="" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299843 2558 flags.go:64] FLAG: --manifest-url-header="" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299847 2558 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299850 2558 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299854 2558 flags.go:64] FLAG: --max-pods="110" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299856 2558 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299859 2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299862 2558 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299865 2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299868 2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 01:16:08.305585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299871 2558 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299874 2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299881 2558 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299884 2558 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299889 2558 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299892 2558 flags.go:64] FLAG: --pod-cidr="" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299895 2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299900 2558 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299903 2558 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299906 2558 flags.go:64] FLAG: --pods-per-core="0" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299909 2558 flags.go:64] FLAG: --port="10250" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299912 2558 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299915 2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-034e2358b12e60eb9" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299918 2558 flags.go:64] FLAG: --qos-reserved="" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299921 2558 flags.go:64] FLAG: --read-only-port="10255" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299924 2558 flags.go:64] FLAG: --register-node="true" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299926 2558 flags.go:64] FLAG: --register-with-taints="" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299931 2558 flags.go:64] FLAG: --registry-burst="10" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299934 2558 flags.go:64] FLAG: --registry-qps="5" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299937 2558 flags.go:64] FLAG: --reserved-cpus="" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299940 2558 flags.go:64] FLAG: --reserved-memory="" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299944 2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299947 2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299950 2558 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299952 2558 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 01:16:08.306212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299955 2558 flags.go:64] FLAG: --runonce="false" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299958 2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299961 2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299964 2558 flags.go:64] FLAG: --seccomp-default="false" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299967 2558 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299969 2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299972 2558 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299975 2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299978 2558 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299981 2558 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299984 2558 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299988 2558 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299990 2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299993 2558 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299996 2558 flags.go:64] FLAG: --system-cgroups="" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.299999 2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.300004 2558 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.300007 2558 flags.go:64] FLAG: --tls-cert-file="" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.300010 2558 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.300014 2558 flags.go:64] FLAG: --tls-min-version="" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.300017 2558 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.300019 2558 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.300022 2558 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.300025 2558 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.300030 2558 flags.go:64] FLAG: --v="2" Sep 16 01:16:08.306850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.300033 2558 flags.go:64] FLAG: --version="false" Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.300042 2558 flags.go:64] FLAG: --vmodule="" Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.300046 2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.300049 2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300156 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300159 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300162 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300165 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300168 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300170 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300173 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300176 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300178 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300181 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300183 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300187 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300191 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300194 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300198 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300201 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 01:16:08.307520 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300204 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300207 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300209 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300212 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300214 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300217 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300219 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300222 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300225 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300227 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300230 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300234 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300236 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300239 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300241 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300245 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300247 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300250 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300252 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300255 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 01:16:08.308019 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300258 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300263 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300267 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300270 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300273 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300276 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300279 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300282 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300285 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300287 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300291 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300295 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300298 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300301 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300304 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300306 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300309 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300312 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300315 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 01:16:08.308530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300317 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300320 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300322 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300325 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300329 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300331 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300334 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300336 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300339 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300342 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300344 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300347 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300349 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300352 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300354 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300357 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300359 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300362 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300364 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300367 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 01:16:08.308993 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300369 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300372 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300374 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300378 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300381 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300383 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300385 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300388 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300391 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300393 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300396 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300398 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300401 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300403 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300406 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300408 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300428 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300430 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300433 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300435 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 01:16:08.309511 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300438 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300440 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300444 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.300447 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.301299 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.307911 2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.307925 2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.307939 2558 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.307948 2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308002 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308007 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308010 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308013 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308016 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308019 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 01:16:08.309996 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308023 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308028 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308032 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308036 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308039 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308042 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308045 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308047 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308050 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308053 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308056 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308058 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308061 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308064 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308066 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308069 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308071 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308074 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308076 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308079 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 01:16:08.310377 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308081 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308084 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308087 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308091 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308093 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308096 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308098 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308101 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308103 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308106 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308108 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308112 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308115 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308118 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308120 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308123 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308125 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308128 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308130 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 01:16:08.310907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308133 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308135 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308138 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308140 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308143 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308145 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308148 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308150 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308152 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308155 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308158 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308160 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308162 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308165 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308167 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308170 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308173 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308177 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308179 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308182 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 01:16:08.311366 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308184 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308187 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308189 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308192 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308194 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308196 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308199 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308201 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308204 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308206 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308209 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308212 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308214 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308217 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308219 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308222 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308224 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308226 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308229 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 01:16:08.311871 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308231 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308234 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308236 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308238 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308241 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308243 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308246 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308248 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308251 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308254 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308256 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308259 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308262 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308265 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308267 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 01:16:08.312318 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.308272 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308376 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308380 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308383 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308386 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308389 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308391 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308394 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308396 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308398 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308401 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308404 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308408 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308434 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308438 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308441 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308443 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308446 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308448 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308451 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 01:16:08.312719 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308454 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308456 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308459 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308461 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308463 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308467 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308469 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308472 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308474 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308478 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308480 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308483 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308485 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308488 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308490 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308493 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308495 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308498 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308500 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308503 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 01:16:08.313212 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308505 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308507 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308510 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308512 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308515 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308517 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308519 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308522 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308524 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308526 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308529 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308531 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308534 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308537 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308539 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308542 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308544 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308547 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308550 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308553 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 01:16:08.313833 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308556 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308559 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308563 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308566 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308569 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308571 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308574 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308576 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308579 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308581 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308583 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308586 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308588 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308590 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308593 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308595 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308597 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308600 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308602 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 01:16:08.314320 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308604 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308607 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308609 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308612 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308614 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308616 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308619 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308621 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308624 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308626 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308629 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308631 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308634 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308637 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308639 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308641 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308644 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308647 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308649 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 01:16:08.314782 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308652 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 01:16:08.315355 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:08.308656 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 01:16:08.315355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.308661 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 01:16:08.315355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.308758 2558 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 01:16:08.315355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.310666 2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 01:16:08.315355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.311724 2558 server.go:1019] "Starting client certificate rotation" Sep 16 01:16:08.315355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.311827 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 01:16:08.315355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.311868 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 01:16:08.315355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.314001 2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 01:16:08.315355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.315115 2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 01:16:08.325926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.325910 2558 log.go:25] "Validated CRI v1 runtime API" Sep 16 01:16:08.331242 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.331221 2558 log.go:25] "Validated CRI v1 image API" Sep 16 01:16:08.332437 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.332396 2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 01:16:08.339770 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.339746 2558 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 93fc2482-5d42-428a-bb0f-78b1578d00fd:/dev/nvme0n1p3 b409ecd6-03f1-44d7-869e-e94cc2eaf7b5:/dev/nvme0n1p4] Sep 16 01:16:08.339843 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.339768 2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 01:16:08.340021 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.340002 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 01:16:08.345979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.345865 2558 manager.go:216] Machine: {Timestamp:2026-09-16 01:16:08.343489927 +0000 UTC m=+0.370657423 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100191 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bd5b2835377da06e28b704b72e2a8 SystemUUID:ec2bd5b2-8353-77da-06e2-8b704b72e2a8 BootID:71b751ec-f9a0-4b24-83e6-e473a0f69403 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:23:28:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:23:28:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:71:7e:b1:97:30 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 01:16:08.345979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.345968 2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 01:16:08.346134 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.346066 2558 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 01:16:08.347277 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.347250 2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 01:16:08.347405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.347280 2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-55.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 01:16:08.348077 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.348067 2558 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 01:16:08.348106 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.348081 2558 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 01:16:08.348106 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.348090 2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 01:16:08.348674 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.348663 2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 01:16:08.349403 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.349394 2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 01:16:08.351947 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.351936 2558 state_mem.go:36] "Initialized new in-memory state store" Sep 16 01:16:08.352057 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.352048 2558 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 01:16:08.352095 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.352077 2558 kubelet.go:486] "Attempting to sync node with API server" Sep 16 01:16:08.352095 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.352087 2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 01:16:08.352155 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.352106 2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 01:16:08.352155 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.352115 2558 kubelet.go:398] "Adding apiserver pod source" Sep 16 01:16:08.352155 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.352123 2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 01:16:08.352499 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.352487 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 01:16:08.352532 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.352514 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 01:16:08.354639 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.354624 2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 01:16:08.356621 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.356608 2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 01:16:08.356674 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.356639 2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 01:16:08.358803 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.358791 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 01:16:08.358884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.358809 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 01:16:08.358884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.358815 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 01:16:08.358884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.358820 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 01:16:08.358884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.358829 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 01:16:08.358884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.358834 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 01:16:08.358884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.358848 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 01:16:08.358884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.358853 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 01:16:08.358884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.358867 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 01:16:08.360007 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.359993 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 01:16:08.360042 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.360017 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 01:16:08.360042 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.360028 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 01:16:08.361010 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.360983 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 01:16:08.361082 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.361065 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 01:16:08.362150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.361840 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 01:16:08.362150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.361856 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 01:16:08.363865 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.363850 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-47ml8" Sep 16 01:16:08.365750 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.365736 2558 server.go:1268] "Started kubelet" Sep 16 01:16:08.365808 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.365783 2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 01:16:08.365906 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.365855 2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 01:16:08.366009 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.365931 2558 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 01:16:08.366518 ip-10-0-137-55 systemd[1]: Started Kubernetes Kubelet. Sep 16 01:16:08.366965 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.366948 2558 server.go:310] "Adding debug handlers to kubelet server" Sep 16 01:16:08.368682 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.367986 2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 01:16:08.370761 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.370745 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-47ml8" Sep 16 01:16:08.371218 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.371200 2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 01:16:08.371455 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.371440 2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-55.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 01:16:08.372424 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.371481 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-55.ec2.internal.18d5a834076643b4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-55.ec2.internal,UID:ip-10-0-137-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-55.ec2.internal,},FirstTimestamp:2026-09-16 01:16:08.365548468 +0000 UTC m=+0.392715964,LastTimestamp:2026-09-16 01:16:08.365548468 +0000 UTC m=+0.392715964,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-55.ec2.internal,}" Sep 16 01:16:08.372853 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.372835 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 01:16:08.373283 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.373264 2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 01:16:08.374144 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.374127 2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 01:16:08.374221 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.374170 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-55.ec2.internal\" not found" Sep 16 01:16:08.374624 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.374599 2558 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 01:16:08.374624 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.374618 2558 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 01:16:08.374840 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.374822 2558 reconstruct.go:97] "Volume reconstruction finished" Sep 16 01:16:08.374840 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.374839 2558 reconciler.go:29] "Reconciler: start to sync state" Sep 16 01:16:08.376104 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.376087 2558 factory.go:55] Registering systemd factory Sep 16 01:16:08.376159 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.376107 2558 factory.go:223] Registration of the systemd container factory successfully Sep 16 01:16:08.376463 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.376450 2558 factory.go:153] Registering CRI-O factory Sep 16 01:16:08.376531 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.376464 2558 factory.go:223] Registration of the crio container factory successfully Sep 16 01:16:08.376634 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.376557 2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 01:16:08.376634 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.376623 2558 factory.go:103] Registering Raw factory Sep 16 01:16:08.376737 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.376645 2558 manager.go:1196] Started watching for new ooms in manager Sep 16 01:16:08.377291 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.377279 2558 manager.go:349] Starting recovery of all containers Sep 16 01:16:08.377593 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.377578 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 01:16:08.381146 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.381121 2558 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-55.ec2.internal\" not found" node="ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.383218 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.383185 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 01:16:08.389778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.389763 2558 manager.go:354] Recovery completed Sep 16 01:16:08.393738 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.393726 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:16:08.395501 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.395487 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:16:08.395579 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.395516 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:16:08.395579 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.395529 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:16:08.396149 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.396134 2558 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 01:16:08.396149 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.396148 2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 01:16:08.396229 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.396156 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:16:08.396229 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.396164 2558 state_mem.go:36] "Initialized new in-memory state store" Sep 16 01:16:08.396901 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.396881 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:16:08.396968 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.396917 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:16:08.396968 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.396930 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:16:08.397306 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.397290 2558 policy_none.go:49] "None policy: Start" Sep 16 01:16:08.397306 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.397309 2558 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 01:16:08.397398 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.397319 2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 01:16:08.398319 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.398309 2558 policy_none.go:47] "Start" Sep 16 01:16:08.428969 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.428953 2558 manager.go:337] "Starting Device Plugin manager" Sep 16 01:16:08.429047 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.428987 2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 01:16:08.429047 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.428995 2558 server.go:87] "Starting device plugin registration server" Sep 16 01:16:08.429222 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.429208 2558 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 01:16:08.429269 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.429226 2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 01:16:08.429317 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.429294 2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 01:16:08.429383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.429370 2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 01:16:08.429445 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.429384 2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 01:16:08.430022 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.429897 2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 01:16:08.430109 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.430045 2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-55.ec2.internal\" not found" Sep 16 01:16:08.513227 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.513203 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 01:16:08.514326 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.513253 2558 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 01:16:08.514326 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.513275 2558 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 01:16:08.514326 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.513308 2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 01:16:08.515156 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.515135 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 01:16:08.530254 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.530239 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:16:08.530984 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.530962 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:16:08.531070 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.530991 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:16:08.531070 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.531002 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:16:08.531070 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.531022 2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.536332 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.536318 2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.536386 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.536338 2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-55.ec2.internal\": node \"ip-10-0-137-55.ec2.internal\" not found" Sep 16 01:16:08.550730 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.550687 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-55.ec2.internal\" not found" Sep 16 01:16:08.613681 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.613658 2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-55.ec2.internal"] Sep 16 01:16:08.613753 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.613716 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:16:08.614517 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.614501 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:16:08.614583 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.614532 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:16:08.614583 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.614542 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:16:08.615895 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.615884 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:16:08.616008 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.615996 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.616043 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.616027 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:16:08.616583 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.616561 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:16:08.616583 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.616580 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:16:08.616692 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.616595 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:16:08.616692 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.616602 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:16:08.616692 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.616616 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:16:08.616692 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.616606 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:16:08.617914 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.617901 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.617982 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.617925 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:16:08.620257 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.620239 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:16:08.620351 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.620268 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:16:08.620351 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.620285 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:16:08.648978 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.648957 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-55.ec2.internal\" not found" node="ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.650844 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.650828 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-55.ec2.internal\" not found" Sep 16 01:16:08.653245 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.653231 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-55.ec2.internal\" not found" node="ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.675918 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.675900 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33cb9aa4ae4df5bb079c09bb4c7f9600-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal\" (UID: \"33cb9aa4ae4df5bb079c09bb4c7f9600\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.676003 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.675926 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33cb9aa4ae4df5bb079c09bb4c7f9600-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal\" (UID: \"33cb9aa4ae4df5bb079c09bb4c7f9600\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.676003 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.675941 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1e7172446b6553eb9092c5ca4da52a1b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-55.ec2.internal\" (UID: \"1e7172446b6553eb9092c5ca4da52a1b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.751178 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.751160 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-55.ec2.internal\" not found" Sep 16 01:16:08.776492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.776467 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33cb9aa4ae4df5bb079c09bb4c7f9600-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal\" (UID: \"33cb9aa4ae4df5bb079c09bb4c7f9600\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.776548 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.776500 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1e7172446b6553eb9092c5ca4da52a1b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-55.ec2.internal\" (UID: \"1e7172446b6553eb9092c5ca4da52a1b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.776548 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.776539 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33cb9aa4ae4df5bb079c09bb4c7f9600-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal\" (UID: \"33cb9aa4ae4df5bb079c09bb4c7f9600\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.776623 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.776575 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33cb9aa4ae4df5bb079c09bb4c7f9600-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal\" (UID: \"33cb9aa4ae4df5bb079c09bb4c7f9600\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.776623 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.776581 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1e7172446b6553eb9092c5ca4da52a1b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-55.ec2.internal\" (UID: \"1e7172446b6553eb9092c5ca4da52a1b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.776623 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.776576 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33cb9aa4ae4df5bb079c09bb4c7f9600-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal\" (UID: \"33cb9aa4ae4df5bb079c09bb4c7f9600\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.851738 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.851691 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-55.ec2.internal\" not found" Sep 16 01:16:08.951215 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.951198 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal" Sep 16 01:16:08.952238 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:08.952222 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-55.ec2.internal\" not found" Sep 16 01:16:08.956789 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:08.956776 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-55.ec2.internal" Sep 16 01:16:09.052784 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:09.052758 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-55.ec2.internal\" not found" Sep 16 01:16:09.153246 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:09.153194 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-55.ec2.internal\" not found" Sep 16 01:16:09.235086 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.235067 2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 01:16:09.253714 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:09.253694 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-55.ec2.internal\" not found" Sep 16 01:16:09.311260 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.311237 2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 01:16:09.311632 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.311342 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 01:16:09.311632 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.311392 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 01:16:09.311632 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.311392 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 01:16:09.354693 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:09.354670 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-55.ec2.internal\" not found" Sep 16 01:16:09.373356 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.373317 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 01:11:10 +0000 UTC" deadline="2028-06-29 04:18:36.187583428 +0000 UTC" Sep 16 01:16:09.373356 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.373351 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15651h2m26.814234785s" Sep 16 01:16:09.454875 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:09.454849 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-55.ec2.internal\" not found" Sep 16 01:16:09.506235 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:09.506208 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33cb9aa4ae4df5bb079c09bb4c7f9600.slice/crio-5ede428d8430e0fda24ecc0fa7a74f89f521d98ef483261bee6e25ff3e05f9b4 WatchSource:0}: Error finding container 5ede428d8430e0fda24ecc0fa7a74f89f521d98ef483261bee6e25ff3e05f9b4: Status 404 returned error can't find the container with id 5ede428d8430e0fda24ecc0fa7a74f89f521d98ef483261bee6e25ff3e05f9b4 Sep 16 01:16:09.506505 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:09.506478 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7172446b6553eb9092c5ca4da52a1b.slice/crio-6d4ceef24ff8d2c2d4d9770067f5c286016d8e0907c84a8dc75487ab11d1b17f WatchSource:0}: Error finding container 6d4ceef24ff8d2c2d4d9770067f5c286016d8e0907c84a8dc75487ab11d1b17f: Status 404 returned error can't find the container with id 6d4ceef24ff8d2c2d4d9770067f5c286016d8e0907c84a8dc75487ab11d1b17f Sep 16 01:16:09.513571 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.513552 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 01:16:09.516096 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.516060 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-55.ec2.internal" event={"ID":"1e7172446b6553eb9092c5ca4da52a1b","Type":"ContainerStarted","Data":"6d4ceef24ff8d2c2d4d9770067f5c286016d8e0907c84a8dc75487ab11d1b17f"} Sep 16 01:16:09.516947 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.516923 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal" event={"ID":"33cb9aa4ae4df5bb079c09bb4c7f9600","Type":"ContainerStarted","Data":"5ede428d8430e0fda24ecc0fa7a74f89f521d98ef483261bee6e25ff3e05f9b4"} Sep 16 01:16:09.537177 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.537156 2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 01:16:09.574568 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.574549 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal" Sep 16 01:16:09.584739 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.584723 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 01:16:09.585732 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.585721 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-55.ec2.internal" Sep 16 01:16:09.593580 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:09.593568 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 01:16:12.001506 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.001363 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 01:16:12.128247 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.128218 2558 apiserver.go:52] "Watching apiserver" Sep 16 01:16:12.136849 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.136826 2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 01:16:12.137278 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.137247 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-6ndbn","kube-system/konnectivity-agent-49bsb","kube-system/kube-apiserver-proxy-ip-10-0-137-55.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s","openshift-image-registry/node-ca-8s8qx","openshift-ovn-kubernetes/ovnkube-node-mgv65","openshift-cluster-node-tuning-operator/tuned-lwwlp","openshift-dns/node-resolver-9twcf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal","openshift-multus/multus-additional-cni-plugins-fjqv9","openshift-multus/multus-rq7cg","openshift-multus/network-metrics-daemon-hs6d5","openshift-network-diagnostics/network-check-target-q6xx7","openshift-network-operator/iptables-alerter-s9jlq"] Sep 16 01:16:12.138605 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.138580 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.140447 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.140430 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:12.140550 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.140498 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:12.140691 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.140667 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f7cx7\"" Sep 16 01:16:12.140956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.140937 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 01:16:12.141054 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.140940 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 01:16:12.142801 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.142782 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s9jlq" Sep 16 01:16:12.144460 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.144239 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.144802 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.144779 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 01:16:12.145418 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.145173 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 01:16:12.145418 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.145218 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 01:16:12.145418 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.145229 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jmhnr\"" Sep 16 01:16:12.147622 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.146383 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 01:16:12.147622 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.146624 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 01:16:12.147622 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.146731 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 01:16:12.147622 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.146737 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-chz9k\"" Sep 16 01:16:12.147622 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.146921 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8s8qx" Sep 16 01:16:12.148053 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.148034 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.148583 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.148564 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 01:16:12.148889 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.148873 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 01:16:12.148963 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.148945 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 01:16:12.149025 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.148881 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 01:16:12.149025 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.149021 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bxwm5\"" Sep 16 01:16:12.149456 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.149435 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9twcf" Sep 16 01:16:12.149741 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.149720 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 01:16:12.150621 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.150601 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 01:16:12.150743 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.150719 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 01:16:12.150975 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.150958 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 01:16:12.151179 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.151146 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:12.151258 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.151221 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:12.152101 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.152084 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-49bsb" Sep 16 01:16:12.153180 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.152949 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 01:16:12.153180 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.153042 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8s45n\"" Sep 16 01:16:12.153180 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.153064 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 01:16:12.153180 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.153138 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 01:16:12.153413 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.153270 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 01:16:12.153471 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.153417 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-98ht7\"" Sep 16 01:16:12.153961 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.153942 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.154118 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.154094 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 01:16:12.154235 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.154145 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 01:16:12.154674 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.154643 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xnqcw\"" Sep 16 01:16:12.155230 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.155208 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.156419 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.156395 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 01:16:12.156587 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.156567 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:12.156587 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.156636 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:12.156803 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.156758 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 01:16:12.156873 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.156803 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 01:16:12.156873 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.156824 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 01:16:12.156999 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.156965 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x6tx4\"" Sep 16 01:16:12.157305 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.157283 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 01:16:12.157767 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.157747 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 01:16:12.157847 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.157801 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ttcvx\"" Sep 16 01:16:12.160017 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.159992 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l98s7\" (UniqueName: \"kubernetes.io/projected/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-kube-api-access-l98s7\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.160096 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160030 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-socket-dir\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.160096 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160053 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-device-dir\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.160096 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160075 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9h4k\" (UniqueName: \"kubernetes.io/projected/91034144-123a-4673-a8f1-c4d8379e85f4-kube-api-access-x9h4k\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.160262 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160101 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a63a0c6-8521-4839-9955-1d4f7e6ea1a9-host\") pod \"node-ca-8s8qx\" (UID: \"2a63a0c6-8521-4839-9955-1d4f7e6ea1a9\") " pod="openshift-image-registry/node-ca-8s8qx" Sep 16 01:16:12.160262 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160122 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-cni-netd\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.160262 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160192 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c5vc\" (UniqueName: \"kubernetes.io/projected/f6be6ffd-8560-4250-84a4-0741806d4858-kube-api-access-5c5vc\") pod \"node-resolver-9twcf\" (UID: \"f6be6ffd-8560-4250-84a4-0741806d4858\") " pod="openshift-dns/node-resolver-9twcf" Sep 16 01:16:12.160262 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160238 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-systemd\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.160435 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160273 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-run\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.160435 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160299 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.160435 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160321 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8pvg\" (UniqueName: \"kubernetes.io/projected/2a63a0c6-8521-4839-9955-1d4f7e6ea1a9-kube-api-access-l8pvg\") pod \"node-ca-8s8qx\" (UID: \"2a63a0c6-8521-4839-9955-1d4f7e6ea1a9\") " pod="openshift-image-registry/node-ca-8s8qx" Sep 16 01:16:12.160435 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160345 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-run-netns\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.160435 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160385 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-run-ovn\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.160435 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160407 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-var-lib-kubelet\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.160697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160435 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-sys-fs\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.160697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160456 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-log-socket\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.160697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160478 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-sysconfig\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.160697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160511 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-host\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.160697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160539 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.160697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160565 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5aa14498-9f6c-43b6-9651-1540143277b1-host-slash\") pod \"iptables-alerter-s9jlq\" (UID: \"5aa14498-9f6c-43b6-9651-1540143277b1\") " pod="openshift-network-operator/iptables-alerter-s9jlq" Sep 16 01:16:12.160697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160587 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-systemd-units\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.160697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160609 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-slash\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.160697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160631 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-cni-bin\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.160697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160653 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-env-overrides\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.160697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160683 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-sysctl-d\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160706 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-sysctl-conf\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160732 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-lib-modules\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160758 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4shr9\" (UniqueName: \"kubernetes.io/projected/5aa14498-9f6c-43b6-9651-1540143277b1-kube-api-access-4shr9\") pod \"iptables-alerter-s9jlq\" (UID: \"5aa14498-9f6c-43b6-9651-1540143277b1\") " pod="openshift-network-operator/iptables-alerter-s9jlq" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160782 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-var-lib-openvswitch\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160804 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-etc-openvswitch\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160835 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160858 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-ovnkube-config\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160879 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-modprobe-d\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160902 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjq9j\" (UniqueName: \"kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j\") pod \"network-check-target-q6xx7\" (UID: \"afe22dac-90f9-4893-9e84-0d7731b9fd8f\") " pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160925 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-registration-dir\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160946 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a63a0c6-8521-4839-9955-1d4f7e6ea1a9-serviceca\") pod \"node-ca-8s8qx\" (UID: \"2a63a0c6-8521-4839-9955-1d4f7e6ea1a9\") " pod="openshift-image-registry/node-ca-8s8qx" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160968 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-run-systemd\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.160991 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-run-openvswitch\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.161012 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-node-log\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.161035 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.161238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.161059 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-sys\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.161778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.161081 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-tuned\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.161778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.161101 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32fa85f9-ec45-4dab-85d9-421aa25754f5-tmp\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.161778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.161124 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5aa14498-9f6c-43b6-9651-1540143277b1-iptables-alerter-script\") pod \"iptables-alerter-s9jlq\" (UID: \"5aa14498-9f6c-43b6-9651-1540143277b1\") " pod="openshift-network-operator/iptables-alerter-s9jlq" Sep 16 01:16:12.161778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.161146 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-kubelet\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.161778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.161192 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6be6ffd-8560-4250-84a4-0741806d4858-hosts-file\") pod \"node-resolver-9twcf\" (UID: \"f6be6ffd-8560-4250-84a4-0741806d4858\") " pod="openshift-dns/node-resolver-9twcf" Sep 16 01:16:12.161778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.161216 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6be6ffd-8560-4250-84a4-0741806d4858-tmp-dir\") pod \"node-resolver-9twcf\" (UID: \"f6be6ffd-8560-4250-84a4-0741806d4858\") " pod="openshift-dns/node-resolver-9twcf" Sep 16 01:16:12.161778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.161240 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-kubernetes\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.161778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.161277 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzmrb\" (UniqueName: \"kubernetes.io/projected/32fa85f9-ec45-4dab-85d9-421aa25754f5-kube-api-access-hzmrb\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.161778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.161310 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-ovn-node-metrics-cert\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.161778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.161334 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-ovnkube-script-lib\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.164605 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.164585 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 01:16:12.183611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.181296 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nrmcv" Sep 16 01:16:12.186805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.186786 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nrmcv" Sep 16 01:16:12.250450 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.250311 2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 01:16:12.262286 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262264 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-socket-dir\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.262397 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262291 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-device-dir\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.262397 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262307 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9h4k\" (UniqueName: \"kubernetes.io/projected/91034144-123a-4673-a8f1-c4d8379e85f4-kube-api-access-x9h4k\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.262397 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262323 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-systemd\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.262397 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262338 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-run\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.262397 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262355 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.262397 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262374 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8pvg\" (UniqueName: \"kubernetes.io/projected/2a63a0c6-8521-4839-9955-1d4f7e6ea1a9-kube-api-access-l8pvg\") pod \"node-ca-8s8qx\" (UID: \"2a63a0c6-8521-4839-9955-1d4f7e6ea1a9\") " pod="openshift-image-registry/node-ca-8s8qx" Sep 16 01:16:12.262670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262399 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-device-dir\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.262670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262420 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-systemd\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.262670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262442 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-socket-dir\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.262670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262486 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.262670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262531 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-system-cni-dir\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.262670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262541 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-run\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.262670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262567 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-run-k8s-cni-cncf-io\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.262670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262625 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-var-lib-cni-multus\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.262670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262653 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-log-socket\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262688 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-log-socket\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262712 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b2afed4-62a6-41d3-b80b-61e37294be11-system-cni-dir\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262730 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hlcn\" (UniqueName: \"kubernetes.io/projected/0b2afed4-62a6-41d3-b80b-61e37294be11-kube-api-access-6hlcn\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262766 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5aa14498-9f6c-43b6-9651-1540143277b1-host-slash\") pod \"iptables-alerter-s9jlq\" (UID: \"5aa14498-9f6c-43b6-9651-1540143277b1\") " pod="openshift-network-operator/iptables-alerter-s9jlq" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262806 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b2afed4-62a6-41d3-b80b-61e37294be11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262817 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5aa14498-9f6c-43b6-9651-1540143277b1-host-slash\") pod \"iptables-alerter-s9jlq\" (UID: \"5aa14498-9f6c-43b6-9651-1540143277b1\") " pod="openshift-network-operator/iptables-alerter-s9jlq" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262828 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shr9\" (UniqueName: \"kubernetes.io/projected/5aa14498-9f6c-43b6-9651-1540143277b1-kube-api-access-4shr9\") pod \"iptables-alerter-s9jlq\" (UID: \"5aa14498-9f6c-43b6-9651-1540143277b1\") " pod="openshift-network-operator/iptables-alerter-s9jlq" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262865 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-var-lib-openvswitch\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262890 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-var-lib-openvswitch\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262892 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-etc-openvswitch\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262920 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262948 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-hostroot\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262963 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-etc-openvswitch\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.262975 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263002 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b2afed4-62a6-41d3-b80b-61e37294be11-cnibin\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263008 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.263076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263030 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-modprobe-d\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263058 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjq9j\" (UniqueName: \"kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j\") pod \"network-check-target-q6xx7\" (UID: \"afe22dac-90f9-4893-9e84-0d7731b9fd8f\") " pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263089 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-registration-dir\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263114 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6be6ffd-8560-4250-84a4-0741806d4858-tmp-dir\") pod \"node-resolver-9twcf\" (UID: \"f6be6ffd-8560-4250-84a4-0741806d4858\") " pod="openshift-dns/node-resolver-9twcf" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263139 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-cnibin\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263184 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-registration-dir\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263210 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-modprobe-d\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263267 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-multus-conf-dir\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263295 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-sys\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263321 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-tuned\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263347 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5aa14498-9f6c-43b6-9651-1540143277b1-iptables-alerter-script\") pod \"iptables-alerter-s9jlq\" (UID: \"5aa14498-9f6c-43b6-9651-1540143277b1\") " pod="openshift-network-operator/iptables-alerter-s9jlq" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263359 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-sys\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263372 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-kubelet\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263398 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6be6ffd-8560-4250-84a4-0741806d4858-hosts-file\") pod \"node-resolver-9twcf\" (UID: \"f6be6ffd-8560-4250-84a4-0741806d4858\") " pod="openshift-dns/node-resolver-9twcf" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263429 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b2afed4-62a6-41d3-b80b-61e37294be11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263456 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0b2afed4-62a6-41d3-b80b-61e37294be11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263477 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-kubelet\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.263864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263484 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvwj5\" (UniqueName: \"kubernetes.io/projected/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-kube-api-access-vvwj5\") pod \"network-metrics-daemon-hs6d5\" (UID: \"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c\") " pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263497 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6be6ffd-8560-4250-84a4-0741806d4858-hosts-file\") pod \"node-resolver-9twcf\" (UID: \"f6be6ffd-8560-4250-84a4-0741806d4858\") " pod="openshift-dns/node-resolver-9twcf" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263513 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6be6ffd-8560-4250-84a4-0741806d4858-tmp-dir\") pod \"node-resolver-9twcf\" (UID: \"f6be6ffd-8560-4250-84a4-0741806d4858\") " pod="openshift-dns/node-resolver-9twcf" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263511 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzmrb\" (UniqueName: \"kubernetes.io/projected/32fa85f9-ec45-4dab-85d9-421aa25754f5-kube-api-access-hzmrb\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263565 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-ovn-node-metrics-cert\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263593 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l98s7\" (UniqueName: \"kubernetes.io/projected/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-kube-api-access-l98s7\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263617 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a63a0c6-8521-4839-9955-1d4f7e6ea1a9-host\") pod \"node-ca-8s8qx\" (UID: \"2a63a0c6-8521-4839-9955-1d4f7e6ea1a9\") " pod="openshift-image-registry/node-ca-8s8qx" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263630 2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263643 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-cni-netd\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263687 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c5vc\" (UniqueName: \"kubernetes.io/projected/f6be6ffd-8560-4250-84a4-0741806d4858-kube-api-access-5c5vc\") pod \"node-resolver-9twcf\" (UID: \"f6be6ffd-8560-4250-84a4-0741806d4858\") " pod="openshift-dns/node-resolver-9twcf" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263702 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a63a0c6-8521-4839-9955-1d4f7e6ea1a9-host\") pod \"node-ca-8s8qx\" (UID: \"2a63a0c6-8521-4839-9955-1d4f7e6ea1a9\") " pod="openshift-image-registry/node-ca-8s8qx" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263714 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-cni-binary-copy\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263740 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-etc-kubernetes\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263768 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-run-netns\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263801 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-run-ovn\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263846 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-cni-netd\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263851 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-var-lib-kubelet\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263913 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-var-lib-kubelet\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.264608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263915 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5aa14498-9f6c-43b6-9651-1540143277b1-iptables-alerter-script\") pod \"iptables-alerter-s9jlq\" (UID: \"5aa14498-9f6c-43b6-9651-1540143277b1\") " pod="openshift-network-operator/iptables-alerter-s9jlq" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263944 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-sys-fs\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263971 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-run-netns\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263976 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-run-ovn\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.263995 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5826c82e-35c3-40b5-a3f8-774437001cf8-dbus\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264013 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-var-lib-kubelet\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264021 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-sysconfig\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264030 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-run-netns\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264046 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-host\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264044 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-sys-fs\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264084 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264112 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-systemd-units\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264127 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-host\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264126 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-sysconfig\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264132 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-slash\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264178 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-slash\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264190 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-systemd-units\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264191 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-cni-bin\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.265430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264221 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-env-overrides\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264244 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r74jm\" (UniqueName: \"kubernetes.io/projected/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-kube-api-access-r74jm\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264244 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91034144-123a-4673-a8f1-c4d8379e85f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264267 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-sysctl-d\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264281 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-cni-bin\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264298 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-sysctl-conf\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264324 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-lib-modules\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264348 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-ovnkube-config\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264366 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-sysctl-d\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264371 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86bd0cbd-70d8-47c2-b865-836ff085681a-agent-certs\") pod \"konnectivity-agent-49bsb\" (UID: \"86bd0cbd-70d8-47c2-b865-836ff085681a\") " pod="kube-system/konnectivity-agent-49bsb" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264405 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-run-multus-certs\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264473 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-lib-modules\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264476 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-sysctl-conf\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264512 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a63a0c6-8521-4839-9955-1d4f7e6ea1a9-serviceca\") pod \"node-ca-8s8qx\" (UID: \"2a63a0c6-8521-4839-9955-1d4f7e6ea1a9\") " pod="openshift-image-registry/node-ca-8s8qx" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264548 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-run-systemd\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264572 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-run-openvswitch\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264596 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-node-log\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.265956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264633 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264662 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-multus-cni-dir\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264669 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-env-overrides\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264686 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b2afed4-62a6-41d3-b80b-61e37294be11-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264716 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-run-openvswitch\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264746 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32fa85f9-ec45-4dab-85d9-421aa25754f5-tmp\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264767 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-multus-socket-dir-parent\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264794 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-var-lib-cni-bin\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264837 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264848 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-run-systemd\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264880 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-node-log\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264923 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-ovnkube-config\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264922 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-multus-daemon-config\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264966 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5826c82e-35c3-40b5-a3f8-774437001cf8-kubelet-config\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.264999 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs\") pod \"network-metrics-daemon-hs6d5\" (UID: \"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c\") " pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.265004 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a63a0c6-8521-4839-9955-1d4f7e6ea1a9-serviceca\") pod \"node-ca-8s8qx\" (UID: \"2a63a0c6-8521-4839-9955-1d4f7e6ea1a9\") " pod="openshift-image-registry/node-ca-8s8qx" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.265032 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-kubernetes\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.266528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.265076 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-kubernetes\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.267005 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.265088 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-ovnkube-script-lib\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.267005 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.265124 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86bd0cbd-70d8-47c2-b865-836ff085681a-konnectivity-ca\") pod \"konnectivity-agent-49bsb\" (UID: \"86bd0cbd-70d8-47c2-b865-836ff085681a\") " pod="kube-system/konnectivity-agent-49bsb" Sep 16 01:16:12.267005 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.265471 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-ovnkube-script-lib\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.267864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.267843 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32fa85f9-ec45-4dab-85d9-421aa25754f5-etc-tuned\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.267976 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.267894 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-ovn-node-metrics-cert\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.268041 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.268012 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32fa85f9-ec45-4dab-85d9-421aa25754f5-tmp\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.271189 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.271149 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:16:12.271290 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.271191 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:16:12.271290 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.271205 2558 projected.go:196] Error preparing data for projected volume kube-api-access-jjq9j for pod openshift-network-diagnostics/network-check-target-q6xx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:12.271746 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.271453 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j podName:afe22dac-90f9-4893-9e84-0d7731b9fd8f nodeName:}" failed. No retries permitted until 2026-09-16 01:16:12.771425779 +0000 UTC m=+3.023848348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jjq9j" (UniqueName: "kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j") pod "network-check-target-q6xx7" (UID: "afe22dac-90f9-4893-9e84-0d7731b9fd8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:12.273069 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.273013 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l98s7\" (UniqueName: \"kubernetes.io/projected/db66eb80-3fa5-4dee-9ba7-c18ba216ab4c-kube-api-access-l98s7\") pod \"ovnkube-node-mgv65\" (UID: \"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.273069 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.273026 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9h4k\" (UniqueName: \"kubernetes.io/projected/91034144-123a-4673-a8f1-c4d8379e85f4-kube-api-access-x9h4k\") pod \"aws-ebs-csi-driver-node-xxj7s\" (UID: \"91034144-123a-4673-a8f1-c4d8379e85f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.273303 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.273136 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8pvg\" (UniqueName: \"kubernetes.io/projected/2a63a0c6-8521-4839-9955-1d4f7e6ea1a9-kube-api-access-l8pvg\") pod \"node-ca-8s8qx\" (UID: \"2a63a0c6-8521-4839-9955-1d4f7e6ea1a9\") " pod="openshift-image-registry/node-ca-8s8qx" Sep 16 01:16:12.273762 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.273738 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzmrb\" (UniqueName: \"kubernetes.io/projected/32fa85f9-ec45-4dab-85d9-421aa25754f5-kube-api-access-hzmrb\") pod \"tuned-lwwlp\" (UID: \"32fa85f9-ec45-4dab-85d9-421aa25754f5\") " pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.275092 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.275068 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4shr9\" (UniqueName: \"kubernetes.io/projected/5aa14498-9f6c-43b6-9651-1540143277b1-kube-api-access-4shr9\") pod \"iptables-alerter-s9jlq\" (UID: \"5aa14498-9f6c-43b6-9651-1540143277b1\") " pod="openshift-network-operator/iptables-alerter-s9jlq" Sep 16 01:16:12.275350 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.275331 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c5vc\" (UniqueName: \"kubernetes.io/projected/f6be6ffd-8560-4250-84a4-0741806d4858-kube-api-access-5c5vc\") pod \"node-resolver-9twcf\" (UID: \"f6be6ffd-8560-4250-84a4-0741806d4858\") " pod="openshift-dns/node-resolver-9twcf" Sep 16 01:16:12.365654 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.365624 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-cni-binary-copy\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.365654 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.365663 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-etc-kubernetes\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.365896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.365689 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-var-lib-kubelet\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.365896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.365739 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-var-lib-kubelet\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.365896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.365741 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-etc-kubernetes\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.365896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.365795 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-run-netns\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.365896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.365834 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5826c82e-35c3-40b5-a3f8-774437001cf8-dbus\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:12.365896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.365866 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r74jm\" (UniqueName: \"kubernetes.io/projected/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-kube-api-access-r74jm\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.365896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.365882 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-run-netns\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.365896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.365894 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86bd0cbd-70d8-47c2-b865-836ff085681a-agent-certs\") pod \"konnectivity-agent-49bsb\" (UID: \"86bd0cbd-70d8-47c2-b865-836ff085681a\") " pod="kube-system/konnectivity-agent-49bsb" Sep 16 01:16:12.366241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.365917 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-run-multus-certs\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.365971 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-run-multus-certs\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.365989 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5826c82e-35c3-40b5-a3f8-774437001cf8-dbus\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:12.366241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366074 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-multus-cni-dir\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366097 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b2afed4-62a6-41d3-b80b-61e37294be11-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.366241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366114 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-multus-socket-dir-parent\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366125 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-multus-cni-dir\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366129 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-var-lib-cni-bin\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366150 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-var-lib-cni-bin\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366198 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-multus-daemon-config\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366212 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-multus-socket-dir-parent\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366232 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5826c82e-35c3-40b5-a3f8-774437001cf8-kubelet-config\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366268 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs\") pod \"network-metrics-daemon-hs6d5\" (UID: \"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c\") " pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366289 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-cni-binary-copy\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366349 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5826c82e-35c3-40b5-a3f8-774437001cf8-kubelet-config\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366296 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86bd0cbd-70d8-47c2-b865-836ff085681a-konnectivity-ca\") pod \"konnectivity-agent-49bsb\" (UID: \"86bd0cbd-70d8-47c2-b865-836ff085681a\") " pod="kube-system/konnectivity-agent-49bsb" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.366383 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366404 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-system-cni-dir\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366428 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-run-k8s-cni-cncf-io\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.366444 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs podName:2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c nodeName:}" failed. No retries permitted until 2026-09-16 01:16:12.866425817 +0000 UTC m=+3.118848387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs") pod "network-metrics-daemon-hs6d5" (UID: "2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366458 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-run-k8s-cni-cncf-io\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366475 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-var-lib-cni-multus\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366498 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b2afed4-62a6-41d3-b80b-61e37294be11-system-cni-dir\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366504 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-system-cni-dir\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366514 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hlcn\" (UniqueName: \"kubernetes.io/projected/0b2afed4-62a6-41d3-b80b-61e37294be11-kube-api-access-6hlcn\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366538 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b2afed4-62a6-41d3-b80b-61e37294be11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366543 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-host-var-lib-cni-multus\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366558 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-hostroot\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.366977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366573 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366591 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b2afed4-62a6-41d3-b80b-61e37294be11-cnibin\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366628 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-cnibin\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366644 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-multus-conf-dir\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366662 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b2afed4-62a6-41d3-b80b-61e37294be11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366678 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0b2afed4-62a6-41d3-b80b-61e37294be11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366686 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-multus-daemon-config\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366695 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvwj5\" (UniqueName: \"kubernetes.io/projected/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-kube-api-access-vvwj5\") pod \"network-metrics-daemon-hs6d5\" (UID: \"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c\") " pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366702 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b2afed4-62a6-41d3-b80b-61e37294be11-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366752 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b2afed4-62a6-41d3-b80b-61e37294be11-system-cni-dir\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366760 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86bd0cbd-70d8-47c2-b865-836ff085681a-konnectivity-ca\") pod \"konnectivity-agent-49bsb\" (UID: \"86bd0cbd-70d8-47c2-b865-836ff085681a\") " pod="kube-system/konnectivity-agent-49bsb" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366765 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b2afed4-62a6-41d3-b80b-61e37294be11-cnibin\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366771 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-multus-conf-dir\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366798 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-hostroot\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366838 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-cnibin\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.366938 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.366994 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b2afed4-62a6-41d3-b80b-61e37294be11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.367805 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.367009 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret podName:5826c82e-35c3-40b5-a3f8-774437001cf8 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:12.86697579 +0000 UTC m=+3.119398372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret") pod "global-pull-secret-syncer-6ndbn" (UID: "5826c82e-35c3-40b5-a3f8-774437001cf8") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:12.368634 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.367174 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b2afed4-62a6-41d3-b80b-61e37294be11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.368634 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.367263 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0b2afed4-62a6-41d3-b80b-61e37294be11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.368871 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.368852 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86bd0cbd-70d8-47c2-b865-836ff085681a-agent-certs\") pod \"konnectivity-agent-49bsb\" (UID: \"86bd0cbd-70d8-47c2-b865-836ff085681a\") " pod="kube-system/konnectivity-agent-49bsb" Sep 16 01:16:12.374577 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.374554 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r74jm\" (UniqueName: \"kubernetes.io/projected/30ce019d-bb0b-45ff-a742-cf9c8f44efe1-kube-api-access-r74jm\") pod \"multus-rq7cg\" (UID: \"30ce019d-bb0b-45ff-a742-cf9c8f44efe1\") " pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.374997 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.374977 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hlcn\" (UniqueName: \"kubernetes.io/projected/0b2afed4-62a6-41d3-b80b-61e37294be11-kube-api-access-6hlcn\") pod \"multus-additional-cni-plugins-fjqv9\" (UID: \"0b2afed4-62a6-41d3-b80b-61e37294be11\") " pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.375189 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.375148 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvwj5\" (UniqueName: \"kubernetes.io/projected/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-kube-api-access-vvwj5\") pod \"network-metrics-daemon-hs6d5\" (UID: \"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c\") " pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:12.453649 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.453578 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" Sep 16 01:16:12.462388 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.462366 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s9jlq" Sep 16 01:16:12.478342 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.478317 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" Sep 16 01:16:12.484042 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.484023 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8s8qx" Sep 16 01:16:12.490703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.490687 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:12.495213 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.495194 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 01:16:12.499869 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.499852 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9twcf" Sep 16 01:16:12.505652 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.505636 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-49bsb" Sep 16 01:16:12.513418 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.513400 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fjqv9" Sep 16 01:16:12.520230 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.520214 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rq7cg" Sep 16 01:16:12.869804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.869769 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:12.869963 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.869810 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjq9j\" (UniqueName: \"kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j\") pod \"network-check-target-q6xx7\" (UID: \"afe22dac-90f9-4893-9e84-0d7731b9fd8f\") " pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:12.869963 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:12.869868 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs\") pod \"network-metrics-daemon-hs6d5\" (UID: \"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c\") " pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:12.869963 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.869904 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:12.869963 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.869945 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:12.869963 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.869967 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret podName:5826c82e-35c3-40b5-a3f8-774437001cf8 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:13.86995004 +0000 UTC m=+4.122372630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret") pod "global-pull-secret-syncer-6ndbn" (UID: "5826c82e-35c3-40b5-a3f8-774437001cf8") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:12.870258 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.869984 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs podName:2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c nodeName:}" failed. No retries permitted until 2026-09-16 01:16:13.869973839 +0000 UTC m=+4.122396406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs") pod "network-metrics-daemon-hs6d5" (UID: "2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:12.870258 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.870000 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:16:12.870258 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.870026 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:16:12.870258 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.870038 2558 projected.go:196] Error preparing data for projected volume kube-api-access-jjq9j for pod openshift-network-diagnostics/network-check-target-q6xx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:12.870258 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:12.870093 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j podName:afe22dac-90f9-4893-9e84-0d7731b9fd8f nodeName:}" failed. No retries permitted until 2026-09-16 01:16:13.870079767 +0000 UTC m=+4.122502334 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjq9j" (UniqueName: "kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j") pod "network-check-target-q6xx7" (UID: "afe22dac-90f9-4893-9e84-0d7731b9fd8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:13.189239 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.189122 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 01:11:12 +0000 UTC" deadline="2028-03-25 10:29:30.739166465 +0000 UTC" Sep 16 01:16:13.189239 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.189187 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13353h13m17.549984645s" Sep 16 01:16:13.278199 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.277498 2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 01:16:13.285886 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:13.285861 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2afed4_62a6_41d3_b80b_61e37294be11.slice/crio-1184b70f1a70d612d83c9a2deccd683181764108e4f266eb2f0977d0706bfa38 WatchSource:0}: Error finding container 1184b70f1a70d612d83c9a2deccd683181764108e4f266eb2f0977d0706bfa38: Status 404 returned error can't find the container with id 1184b70f1a70d612d83c9a2deccd683181764108e4f266eb2f0977d0706bfa38 Sep 16 01:16:13.286522 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:13.286503 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bd0cbd_70d8_47c2_b865_836ff085681a.slice/crio-99ad8bc6c76091dca0eb0f8ba90ef00d5b78a928f946cf22588d7d3295b0025d WatchSource:0}: Error finding container 99ad8bc6c76091dca0eb0f8ba90ef00d5b78a928f946cf22588d7d3295b0025d: Status 404 returned error can't find the container with id 99ad8bc6c76091dca0eb0f8ba90ef00d5b78a928f946cf22588d7d3295b0025d Sep 16 01:16:13.287527 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:13.287466 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91034144_123a_4673_a8f1_c4d8379e85f4.slice/crio-219267ae1a20f593aec76c1d71a1a7ad75eb03a00da329134b1dea40c5fedea6 WatchSource:0}: Error finding container 219267ae1a20f593aec76c1d71a1a7ad75eb03a00da329134b1dea40c5fedea6: Status 404 returned error can't find the container with id 219267ae1a20f593aec76c1d71a1a7ad75eb03a00da329134b1dea40c5fedea6 Sep 16 01:16:13.288388 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.288310 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:13.288388 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:13.288330 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ce019d_bb0b_45ff_a742_cf9c8f44efe1.slice/crio-b39a5b346ac7966d5135ea607dd78a037ebb19c5f9a80e532ddefdfd39f742af WatchSource:0}: Error finding container b39a5b346ac7966d5135ea607dd78a037ebb19c5f9a80e532ddefdfd39f742af: Status 404 returned error can't find the container with id b39a5b346ac7966d5135ea607dd78a037ebb19c5f9a80e532ddefdfd39f742af Sep 16 01:16:13.288518 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:13.288420 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:13.288518 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.288435 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:13.288518 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:13.288507 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:13.290627 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:13.290602 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fa85f9_ec45_4dab_85d9_421aa25754f5.slice/crio-63b6d423aebff2bf80a6e64b224a815fd03e469feb6ed888c88a2783ece8dcb9 WatchSource:0}: Error finding container 63b6d423aebff2bf80a6e64b224a815fd03e469feb6ed888c88a2783ece8dcb9: Status 404 returned error can't find the container with id 63b6d423aebff2bf80a6e64b224a815fd03e469feb6ed888c88a2783ece8dcb9 Sep 16 01:16:13.291459 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:13.291436 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a63a0c6_8521_4839_9955_1d4f7e6ea1a9.slice/crio-acb92129c21aeebdc057d548a00ea8e7e10000f095f01f4c5f368766533fd3c8 WatchSource:0}: Error finding container acb92129c21aeebdc057d548a00ea8e7e10000f095f01f4c5f368766533fd3c8: Status 404 returned error can't find the container with id acb92129c21aeebdc057d548a00ea8e7e10000f095f01f4c5f368766533fd3c8 Sep 16 01:16:13.292851 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:13.292829 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6be6ffd_8560_4250_84a4_0741806d4858.slice/crio-1f2e15fa215b2cee4e7c8ea41403c7674e5439a79c74d638580c377078755927 WatchSource:0}: Error finding container 1f2e15fa215b2cee4e7c8ea41403c7674e5439a79c74d638580c377078755927: Status 404 returned error can't find the container with id 1f2e15fa215b2cee4e7c8ea41403c7674e5439a79c74d638580c377078755927 Sep 16 01:16:13.293570 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:13.293542 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb66eb80_3fa5_4dee_9ba7_c18ba216ab4c.slice/crio-8ececd87ac9493bf01ebc328a991248028888cfdd8e8d7ae61d981b100877afb WatchSource:0}: Error finding container 8ececd87ac9493bf01ebc328a991248028888cfdd8e8d7ae61d981b100877afb: Status 404 returned error can't find the container with id 8ececd87ac9493bf01ebc328a991248028888cfdd8e8d7ae61d981b100877afb Sep 16 01:16:13.300866 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.300838 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8s8qx" event={"ID":"2a63a0c6-8521-4839-9955-1d4f7e6ea1a9","Type":"ContainerStarted","Data":"acb92129c21aeebdc057d548a00ea8e7e10000f095f01f4c5f368766533fd3c8"} Sep 16 01:16:13.302217 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.302195 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rq7cg" event={"ID":"30ce019d-bb0b-45ff-a742-cf9c8f44efe1","Type":"ContainerStarted","Data":"b39a5b346ac7966d5135ea607dd78a037ebb19c5f9a80e532ddefdfd39f742af"} Sep 16 01:16:13.303417 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.303398 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" event={"ID":"91034144-123a-4673-a8f1-c4d8379e85f4","Type":"ContainerStarted","Data":"219267ae1a20f593aec76c1d71a1a7ad75eb03a00da329134b1dea40c5fedea6"} Sep 16 01:16:13.304739 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.304714 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-49bsb" event={"ID":"86bd0cbd-70d8-47c2-b865-836ff085681a","Type":"ContainerStarted","Data":"99ad8bc6c76091dca0eb0f8ba90ef00d5b78a928f946cf22588d7d3295b0025d"} Sep 16 01:16:13.306179 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.306145 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s9jlq" event={"ID":"5aa14498-9f6c-43b6-9651-1540143277b1","Type":"ContainerStarted","Data":"9c95446412e3708b6fe7ba2c33d6d434bc4edc073bb21f90c12eb2f8356ac022"} Sep 16 01:16:13.307501 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.307476 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" event={"ID":"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c","Type":"ContainerStarted","Data":"8ececd87ac9493bf01ebc328a991248028888cfdd8e8d7ae61d981b100877afb"} Sep 16 01:16:13.308799 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.308763 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" event={"ID":"32fa85f9-ec45-4dab-85d9-421aa25754f5","Type":"ContainerStarted","Data":"63b6d423aebff2bf80a6e64b224a815fd03e469feb6ed888c88a2783ece8dcb9"} Sep 16 01:16:13.310228 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.310208 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjqv9" event={"ID":"0b2afed4-62a6-41d3-b80b-61e37294be11","Type":"ContainerStarted","Data":"1184b70f1a70d612d83c9a2deccd683181764108e4f266eb2f0977d0706bfa38"} Sep 16 01:16:13.879841 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.879667 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:13.879990 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.879856 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjq9j\" (UniqueName: \"kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j\") pod \"network-check-target-q6xx7\" (UID: \"afe22dac-90f9-4893-9e84-0d7731b9fd8f\") " pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:13.879990 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:13.879909 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs\") pod \"network-metrics-daemon-hs6d5\" (UID: \"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c\") " pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:13.880101 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:13.880020 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:13.880101 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:13.880076 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs podName:2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c nodeName:}" failed. No retries permitted until 2026-09-16 01:16:15.880057431 +0000 UTC m=+6.132480019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs") pod "network-metrics-daemon-hs6d5" (UID: "2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:13.880958 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:13.880449 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:13.880958 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:13.880493 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret podName:5826c82e-35c3-40b5-a3f8-774437001cf8 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:15.880479507 +0000 UTC m=+6.132902087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret") pod "global-pull-secret-syncer-6ndbn" (UID: "5826c82e-35c3-40b5-a3f8-774437001cf8") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:13.880958 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:13.880559 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:16:13.880958 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:13.880571 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:16:13.880958 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:13.880582 2558 projected.go:196] Error preparing data for projected volume kube-api-access-jjq9j for pod openshift-network-diagnostics/network-check-target-q6xx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:13.880958 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:13.880611 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j podName:afe22dac-90f9-4893-9e84-0d7731b9fd8f nodeName:}" failed. No retries permitted until 2026-09-16 01:16:15.88060185 +0000 UTC m=+6.133024416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjq9j" (UniqueName: "kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j") pod "network-check-target-q6xx7" (UID: "afe22dac-90f9-4893-9e84-0d7731b9fd8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:14.190538 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:14.190490 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 01:11:12 +0000 UTC" deadline="2028-06-14 04:05:05.123846722 +0000 UTC" Sep 16 01:16:14.190538 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:14.190535 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15290h48m50.933314028s" Sep 16 01:16:14.288373 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:14.288347 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:14.288536 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:14.288482 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:14.355925 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:14.355890 2558 generic.go:356] "Generic (PLEG): container finished" podID="33cb9aa4ae4df5bb079c09bb4c7f9600" containerID="4e77ebd7e53ccf12333db6f24b41068a79149b75c8f9ccc7b4453481c25aaa7a" exitCode=0 Sep 16 01:16:14.356060 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:14.355986 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal" event={"ID":"33cb9aa4ae4df5bb079c09bb4c7f9600","Type":"ContainerDied","Data":"4e77ebd7e53ccf12333db6f24b41068a79149b75c8f9ccc7b4453481c25aaa7a"} Sep 16 01:16:14.385412 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:14.381229 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-55.ec2.internal" event={"ID":"1e7172446b6553eb9092c5ca4da52a1b","Type":"ContainerStarted","Data":"21a827d8c26dc96b84790b0615353e8055b1f7cc70b87bf091a7486e263c1a3e"} Sep 16 01:16:14.392374 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:14.392309 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9twcf" event={"ID":"f6be6ffd-8560-4250-84a4-0741806d4858","Type":"ContainerStarted","Data":"1f2e15fa215b2cee4e7c8ea41403c7674e5439a79c74d638580c377078755927"} Sep 16 01:16:15.289388 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:15.288776 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:15.289388 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:15.288886 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:15.289388 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:15.289254 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:15.289388 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:15.289344 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:15.418249 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:15.418210 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal" event={"ID":"33cb9aa4ae4df5bb079c09bb4c7f9600","Type":"ContainerStarted","Data":"8cbf3c6abb7e58479f63adc3b3bf2d85a4bbe3ff97dcf2de5c092a513823b7cf"} Sep 16 01:16:15.434321 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:15.434269 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-55.ec2.internal" podStartSLOduration=4.434249995 podStartE2EDuration="4.434249995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:14.394894021 +0000 UTC m=+4.647316610" watchObservedRunningTime="2026-09-16 01:16:15.434249995 +0000 UTC m=+5.686672586" Sep 16 01:16:15.897176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:15.897118 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:15.897363 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:15.897195 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjq9j\" (UniqueName: \"kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j\") pod \"network-check-target-q6xx7\" (UID: \"afe22dac-90f9-4893-9e84-0d7731b9fd8f\") " pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:15.897363 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:15.897299 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs\") pod \"network-metrics-daemon-hs6d5\" (UID: \"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c\") " pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:15.897473 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:15.897437 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:15.897529 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:15.897502 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs podName:2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c nodeName:}" failed. No retries permitted until 2026-09-16 01:16:19.897482911 +0000 UTC m=+10.149905484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs") pod "network-metrics-daemon-hs6d5" (UID: "2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:15.897592 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:15.897576 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:15.897642 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:15.897618 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret podName:5826c82e-35c3-40b5-a3f8-774437001cf8 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:19.897605859 +0000 UTC m=+10.150028436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret") pod "global-pull-secret-syncer-6ndbn" (UID: "5826c82e-35c3-40b5-a3f8-774437001cf8") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:15.897722 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:15.897706 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:16:15.897775 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:15.897730 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:16:15.897775 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:15.897744 2558 projected.go:196] Error preparing data for projected volume kube-api-access-jjq9j for pod openshift-network-diagnostics/network-check-target-q6xx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:15.897860 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:15.897785 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j podName:afe22dac-90f9-4893-9e84-0d7731b9fd8f nodeName:}" failed. No retries permitted until 2026-09-16 01:16:19.897772112 +0000 UTC m=+10.150194685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjq9j" (UniqueName: "kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j") pod "network-check-target-q6xx7" (UID: "afe22dac-90f9-4893-9e84-0d7731b9fd8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:16.291338 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:16.291255 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:16.291779 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:16.291402 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:17.289073 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:17.288417 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:17.289073 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:17.288640 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:17.289073 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:17.288750 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:17.289073 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:17.288809 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:18.290454 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:18.290420 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:18.290818 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:18.290558 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:19.289148 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:19.288592 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:19.289148 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:19.288694 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:19.289148 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:19.289022 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:19.289148 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:19.289104 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:19.932601 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:19.932557 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs\") pod \"network-metrics-daemon-hs6d5\" (UID: \"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c\") " pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:19.933060 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:19.932627 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:19.933060 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:19.932662 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjq9j\" (UniqueName: \"kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j\") pod \"network-check-target-q6xx7\" (UID: \"afe22dac-90f9-4893-9e84-0d7731b9fd8f\") " pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:19.933060 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:19.932801 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:16:19.933060 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:19.932827 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:16:19.933060 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:19.932837 2558 projected.go:196] Error preparing data for projected volume kube-api-access-jjq9j for pod openshift-network-diagnostics/network-check-target-q6xx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:19.933060 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:19.932882 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j podName:afe22dac-90f9-4893-9e84-0d7731b9fd8f nodeName:}" failed. No retries permitted until 2026-09-16 01:16:27.932868814 +0000 UTC m=+18.185291380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjq9j" (UniqueName: "kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j") pod "network-check-target-q6xx7" (UID: "afe22dac-90f9-4893-9e84-0d7731b9fd8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:19.933060 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:19.932935 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:19.933060 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:19.932958 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs podName:2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c nodeName:}" failed. No retries permitted until 2026-09-16 01:16:27.932950644 +0000 UTC m=+18.185373211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs") pod "network-metrics-daemon-hs6d5" (UID: "2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:19.933060 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:19.932990 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:19.933060 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:19.933018 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret podName:5826c82e-35c3-40b5-a3f8-774437001cf8 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:27.93300704 +0000 UTC m=+18.185429607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret") pod "global-pull-secret-syncer-6ndbn" (UID: "5826c82e-35c3-40b5-a3f8-774437001cf8") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:20.293355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:20.293269 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:20.293498 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:20.293401 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:21.288752 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:21.288718 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:21.289244 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:21.288849 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:21.289305 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:21.289278 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:21.289414 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:21.289389 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:22.292311 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:22.292281 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:22.292774 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:22.292412 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:23.289073 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:23.289039 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:23.289253 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:23.289040 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:23.289253 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:23.289195 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:23.289253 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:23.289233 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:24.288646 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:24.288563 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:24.289042 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:24.288684 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:25.289247 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:25.289205 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:25.289247 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:25.289245 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:25.289715 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:25.289331 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:25.289715 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:25.289454 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:26.288408 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:26.288370 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:26.288608 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:26.288523 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:27.289153 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:27.289094 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:27.289580 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:27.289240 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:27.289580 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:27.289109 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:27.289580 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:27.289389 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:27.994529 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:27.994493 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs\") pod \"network-metrics-daemon-hs6d5\" (UID: \"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c\") " pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:27.994718 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:27.994540 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:27.994718 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:27.994565 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjq9j\" (UniqueName: \"kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j\") pod \"network-check-target-q6xx7\" (UID: \"afe22dac-90f9-4893-9e84-0d7731b9fd8f\") " pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:27.994718 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:27.994706 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:27.994877 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:27.994732 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:27.994877 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:27.994750 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:16:27.994877 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:27.994770 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:16:27.994877 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:27.994776 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs podName:2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c nodeName:}" failed. No retries permitted until 2026-09-16 01:16:43.994759632 +0000 UTC m=+34.247182202 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs") pod "network-metrics-daemon-hs6d5" (UID: "2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:27.994877 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:27.994785 2558 projected.go:196] Error preparing data for projected volume kube-api-access-jjq9j for pod openshift-network-diagnostics/network-check-target-q6xx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:27.994877 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:27.994793 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret podName:5826c82e-35c3-40b5-a3f8-774437001cf8 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:43.994785878 +0000 UTC m=+34.247208445 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret") pod "global-pull-secret-syncer-6ndbn" (UID: "5826c82e-35c3-40b5-a3f8-774437001cf8") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:27.994877 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:27.994833 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j podName:afe22dac-90f9-4893-9e84-0d7731b9fd8f nodeName:}" failed. No retries permitted until 2026-09-16 01:16:43.994818577 +0000 UTC m=+34.247241144 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjq9j" (UniqueName: "kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j") pod "network-check-target-q6xx7" (UID: "afe22dac-90f9-4893-9e84-0d7731b9fd8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:28.288841 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:28.288761 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:28.288989 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:28.288898 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:29.288573 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:29.288540 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:29.289003 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:29.288580 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:29.289003 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:29.288660 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:29.289003 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:29.288818 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:30.289194 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:30.289150 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:30.289631 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:30.289279 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:31.288309 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:31.288286 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:31.288421 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:31.288401 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:31.288476 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:31.288448 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:31.288532 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:31.288514 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:31.469568 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:31.469533 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-49bsb" event={"ID":"86bd0cbd-70d8-47c2-b865-836ff085681a","Type":"ContainerStarted","Data":"65ca60d97038b0c464b16e039a18067c3e8b2b6c2c60dac9bdd934ba070a20cc"} Sep 16 01:16:31.476104 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:31.475970 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" event={"ID":"32fa85f9-ec45-4dab-85d9-421aa25754f5","Type":"ContainerStarted","Data":"e6b5e0c05db2c323f57892b75351e88a0cdba2defe4439405188ad14e62a57e9"} Sep 16 01:16:31.484959 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:31.484685 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-55.ec2.internal" podStartSLOduration=20.484665863 podStartE2EDuration="20.484665863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:15.43484346 +0000 UTC m=+5.687266049" watchObservedRunningTime="2026-09-16 01:16:31.484665863 +0000 UTC m=+21.737088453" Sep 16 01:16:31.485319 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:31.485233 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-49bsb" podStartSLOduration=7.315392029 podStartE2EDuration="21.485218674s" totalImagesPullingTime="14.169826645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:31.484343575 +0000 UTC m=+21.736766164" watchObservedRunningTime="2026-09-16 01:16:31.485218674 +0000 UTC m=+21.737641264" Sep 16 01:16:31.501817 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:31.501246 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lwwlp" podStartSLOduration=3.505738652 podStartE2EDuration="21.501226628s" totalImagesPullingTime="17.995487976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:31.500053766 +0000 UTC m=+21.752476356" watchObservedRunningTime="2026-09-16 01:16:31.501226628 +0000 UTC m=+21.753649222" Sep 16 01:16:32.289287 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.289029 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:32.289555 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:32.289419 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:32.295923 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.295894 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-49bsb" Sep 16 01:16:32.296040 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.295996 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-49bsb" Sep 16 01:16:32.479760 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.479720 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8s8qx" event={"ID":"2a63a0c6-8521-4839-9955-1d4f7e6ea1a9","Type":"ContainerStarted","Data":"f5c8b8caa77abcc334fd2d161357e70fa46d5028a4a8e19477341c58fc177856"} Sep 16 01:16:32.481323 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.481296 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rq7cg" event={"ID":"30ce019d-bb0b-45ff-a742-cf9c8f44efe1","Type":"ContainerStarted","Data":"7538f92a08d3d28c0048c9bbf988c359824f5bf24577a0d6e0052f7962405253"} Sep 16 01:16:32.482940 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.482910 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" event={"ID":"91034144-123a-4673-a8f1-c4d8379e85f4","Type":"ContainerStarted","Data":"d0df1e2e4138aa6f8f23b46ee415da96fb0b4f4d88c0f5b81dd16c87c8162a67"} Sep 16 01:16:32.484367 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.484342 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9twcf" event={"ID":"f6be6ffd-8560-4250-84a4-0741806d4858","Type":"ContainerStarted","Data":"e31d379425b55f5be740656acb34b562a6462ee3d06f1a9f21d97001b8c387f7"} Sep 16 01:16:32.487023 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.487005 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" event={"ID":"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c","Type":"ContainerStarted","Data":"07ba3a07454ec852e8a51b574d0c597e798517ad989bce41a90a083aee41ca30"} Sep 16 01:16:32.487127 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.487061 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" event={"ID":"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c","Type":"ContainerStarted","Data":"71f530391d7b91cc1268e8f10e172d77a13b2ed2bf8ccb1def02bfabeec561e3"} Sep 16 01:16:32.487127 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.487092 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" event={"ID":"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c","Type":"ContainerStarted","Data":"351754873d0251c5279a7661e7ad01a83634474cb6de48b006f50817b8caac82"} Sep 16 01:16:32.487127 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.487123 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" event={"ID":"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c","Type":"ContainerStarted","Data":"c814e7a10d6c14e426c769dadf5e68b68ba236a98ea3ca6169c24fbf621dbeb9"} Sep 16 01:16:32.487276 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.487178 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" event={"ID":"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c","Type":"ContainerStarted","Data":"c298dd17ed5b4c969289f72fa9137774b16fd9167b09aab1881175be299fc793"} Sep 16 01:16:32.487276 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.487210 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" event={"ID":"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c","Type":"ContainerStarted","Data":"681ce6577c08a8d11aa4a02cb874292ee8888a2ba5bd057cd17c21e85eac1d31"} Sep 16 01:16:32.488404 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.488381 2558 generic.go:356] "Generic (PLEG): container finished" podID="0b2afed4-62a6-41d3-b80b-61e37294be11" containerID="22fee581482069f590abbde8115498ddac2a1c9f1999b896ecc62d38c6017199" exitCode=0 Sep 16 01:16:32.488477 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.488461 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjqv9" event={"ID":"0b2afed4-62a6-41d3-b80b-61e37294be11","Type":"ContainerDied","Data":"22fee581482069f590abbde8115498ddac2a1c9f1999b896ecc62d38c6017199"} Sep 16 01:16:32.488795 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.488772 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-49bsb" Sep 16 01:16:32.489728 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.489241 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-49bsb" Sep 16 01:16:32.492675 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.492639 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8s8qx" podStartSLOduration=4.513919562 podStartE2EDuration="22.492626276s" totalImagesPullingTime="17.978706714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:32.492371632 +0000 UTC m=+22.744794222" watchObservedRunningTime="2026-09-16 01:16:32.492626276 +0000 UTC m=+22.745048864" Sep 16 01:16:32.505369 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.505327 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9twcf" podStartSLOduration=4.5320094 podStartE2EDuration="22.505314542s" totalImagesPullingTime="17.973305142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:32.505273903 +0000 UTC m=+22.757696492" watchObservedRunningTime="2026-09-16 01:16:32.505314542 +0000 UTC m=+22.757737132" Sep 16 01:16:32.577794 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.577767 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:32.675494 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.675465 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:32.773808 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.773781 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:32.878482 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:32.878458 2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 01:16:33.218811 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:33.218686 2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T01:16:32.878477398Z","UUID":"a0cc79e9-5400-44b9-aceb-db88cbcfa382","Handler":null,"Name":"","Endpoint":""} Sep 16 01:16:33.220011 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:33.219996 2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 01:16:33.220118 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:33.220021 2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 01:16:33.288719 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:33.288685 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:33.288719 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:33.288709 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:33.288924 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:33.288815 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:33.288924 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:33.288894 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:33.492596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:33.492509 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s9jlq" event={"ID":"5aa14498-9f6c-43b6-9651-1540143277b1","Type":"ContainerStarted","Data":"8419c0728c4d00fe22b39c609752e9b2299a6b539acf255be33dd95e632e1088"} Sep 16 01:16:33.494986 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:33.494955 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" event={"ID":"91034144-123a-4673-a8f1-c4d8379e85f4","Type":"ContainerStarted","Data":"09da244b7d0498c75bbb8470a9df817e4389fc26ca8145ccf2a84481b99846d2"} Sep 16 01:16:33.505656 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:33.505612 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rq7cg" podStartSLOduration=5.451977654 podStartE2EDuration="23.505595381s" totalImagesPullingTime="18.053617727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:32.55249526 +0000 UTC m=+22.804917849" watchObservedRunningTime="2026-09-16 01:16:33.505595381 +0000 UTC m=+23.758017973" Sep 16 01:16:33.621927 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:33.621881 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:33.674361 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:33.674329 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:34.288607 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:34.288575 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:34.288774 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:34.288689 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:34.501658 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:34.501623 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" event={"ID":"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c","Type":"ContainerStarted","Data":"499d08a3e6b85f1748870b1e332b5ff303007565fcc7698341b1bed39b2d96c8"} Sep 16 01:16:34.504190 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:34.504143 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" event={"ID":"91034144-123a-4673-a8f1-c4d8379e85f4","Type":"ContainerStarted","Data":"ade5126fef7c1eb6f678cd320c8145f825f06230449ecaacc2e8241aa3cb97da"} Sep 16 01:16:34.525280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:34.525233 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s9jlq" podStartSLOduration=6.526583388 podStartE2EDuration="24.525214383s" totalImagesPullingTime="17.998630995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:33.505120861 +0000 UTC m=+23.757543455" watchObservedRunningTime="2026-09-16 01:16:34.525214383 +0000 UTC m=+24.777636974" Sep 16 01:16:34.525478 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:34.525452 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxj7s" podStartSLOduration=4.368976053 podStartE2EDuration="24.525441902s" totalImagesPullingTime="20.156465849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:16:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:34.525074485 +0000 UTC m=+24.777497096" watchObservedRunningTime="2026-09-16 01:16:34.525441902 +0000 UTC m=+24.777864511" Sep 16 01:16:35.288638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:35.288491 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:35.288794 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:35.288494 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:35.288794 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:35.288706 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:35.288794 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:35.288780 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:36.288278 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:36.288249 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:36.289011 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:36.288394 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:36.513003 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:36.512971 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" event={"ID":"db66eb80-3fa5-4dee-9ba7-c18ba216ab4c","Type":"ContainerStarted","Data":"ca75616f379ace778443559613e0fdd82f3033e8f56783511f9d0af5f0c70ee8"} Sep 16 01:16:36.513432 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:36.513329 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:36.513432 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:36.513401 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:36.531530 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:36.531508 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:36.536662 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:36.536621 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" podStartSLOduration=8.504883721 podStartE2EDuration="26.536604528s" totalImagesPullingTime="18.031720807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:36.535500772 +0000 UTC m=+26.787923363" watchObservedRunningTime="2026-09-16 01:16:36.536604528 +0000 UTC m=+26.789027117" Sep 16 01:16:37.288793 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:37.288761 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:37.289301 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:37.288760 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:37.289301 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:37.288900 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:37.289301 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:37.288949 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:37.526867 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:37.526756 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:37.548887 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:37.548815 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:16:38.076438 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:38.076207 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6ndbn"] Sep 16 01:16:38.076648 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:38.076528 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:38.076648 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:38.076629 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:38.079636 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:38.079612 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hs6d5"] Sep 16 01:16:38.079754 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:38.079741 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:38.079862 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:38.079844 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:38.089178 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:38.089103 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q6xx7"] Sep 16 01:16:38.089298 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:38.089199 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:38.089352 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:38.089291 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:39.288842 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:39.288772 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:39.289235 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:39.288878 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:39.289235 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:39.288905 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:39.289235 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:39.288965 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:39.531915 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:39.531885 2558 generic.go:356] "Generic (PLEG): container finished" podID="0b2afed4-62a6-41d3-b80b-61e37294be11" containerID="bbcab3847783ffdd9eec10b978cdeddf209e5ef26f6f7afa195a593d1061befd" exitCode=0 Sep 16 01:16:39.532054 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:39.531962 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjqv9" event={"ID":"0b2afed4-62a6-41d3-b80b-61e37294be11","Type":"ContainerDied","Data":"bbcab3847783ffdd9eec10b978cdeddf209e5ef26f6f7afa195a593d1061befd"} Sep 16 01:16:40.289815 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:40.289791 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:40.290128 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:40.289878 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:41.288339 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:41.288183 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:41.288489 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:41.288252 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:41.288489 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:41.288414 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:41.288489 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:41.288472 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:41.540557 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:41.540486 2558 generic.go:356] "Generic (PLEG): container finished" podID="0b2afed4-62a6-41d3-b80b-61e37294be11" containerID="a73562326a4bacb3f85902db638a22e2ca3df9f9a266f028b37b82244f48574d" exitCode=0 Sep 16 01:16:41.540557 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:41.540528 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjqv9" event={"ID":"0b2afed4-62a6-41d3-b80b-61e37294be11","Type":"ContainerDied","Data":"a73562326a4bacb3f85902db638a22e2ca3df9f9a266f028b37b82244f48574d"} Sep 16 01:16:42.289252 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:42.289220 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:42.289432 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:42.289345 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:43.289020 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:43.288994 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:43.289020 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:43.289000 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:43.289555 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:43.289096 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6xx7" podUID="afe22dac-90f9-4893-9e84-0d7731b9fd8f" Sep 16 01:16:43.289555 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:43.289175 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs6d5" podUID="2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c" Sep 16 01:16:43.547507 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:43.547447 2558 generic.go:356] "Generic (PLEG): container finished" podID="0b2afed4-62a6-41d3-b80b-61e37294be11" containerID="7edb16aca59c555c3299ea0b50b50658556db1d41f7871d8f4ac57eb42d90488" exitCode=0 Sep 16 01:16:43.547507 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:43.547498 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjqv9" event={"ID":"0b2afed4-62a6-41d3-b80b-61e37294be11","Type":"ContainerDied","Data":"7edb16aca59c555c3299ea0b50b50658556db1d41f7871d8f4ac57eb42d90488"} Sep 16 01:16:44.017564 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:44.017535 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs\") pod \"network-metrics-daemon-hs6d5\" (UID: \"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c\") " pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:44.017767 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:44.017573 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:44.017767 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:44.017591 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjq9j\" (UniqueName: \"kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j\") pod \"network-check-target-q6xx7\" (UID: \"afe22dac-90f9-4893-9e84-0d7731b9fd8f\") " pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:44.017767 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:44.017670 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:44.017767 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:44.017675 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:44.017767 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:44.017689 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:16:44.017767 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:44.017703 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:16:44.017767 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:44.017714 2558 projected.go:196] Error preparing data for projected volume kube-api-access-jjq9j for pod openshift-network-diagnostics/network-check-target-q6xx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:44.017767 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:44.017722 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs podName:2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c nodeName:}" failed. No retries permitted until 2026-09-16 01:17:16.017708471 +0000 UTC m=+66.270131037 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs") pod "network-metrics-daemon-hs6d5" (UID: "2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:44.017767 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:44.017735 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret podName:5826c82e-35c3-40b5-a3f8-774437001cf8 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:16.017728669 +0000 UTC m=+66.270151236 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret") pod "global-pull-secret-syncer-6ndbn" (UID: "5826c82e-35c3-40b5-a3f8-774437001cf8") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:44.017767 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:44.017751 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j podName:afe22dac-90f9-4893-9e84-0d7731b9fd8f nodeName:}" failed. No retries permitted until 2026-09-16 01:17:16.017740748 +0000 UTC m=+66.270163315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjq9j" (UniqueName: "kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j") pod "network-check-target-q6xx7" (UID: "afe22dac-90f9-4893-9e84-0d7731b9fd8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:44.288474 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:44.288406 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:44.288649 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:44.288526 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ndbn" podUID="5826c82e-35c3-40b5-a3f8-774437001cf8" Sep 16 01:16:44.983195 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:44.983149 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-55.ec2.internal" event="NodeReady" Sep 16 01:16:44.983665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:44.983277 2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 01:16:45.030099 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.030071 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lnb9f"] Sep 16 01:16:45.054312 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.054278 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zczs4"] Sep 16 01:16:45.054490 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.054470 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:45.056879 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.056856 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 01:16:45.057004 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.056988 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 01:16:45.057068 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.056983 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sq7hb\"" Sep 16 01:16:45.070014 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.069955 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lnb9f"] Sep 16 01:16:45.070014 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.069995 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zczs4"] Sep 16 01:16:45.070184 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.070085 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zczs4" Sep 16 01:16:45.072123 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.072101 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tgqvm\"" Sep 16 01:16:45.072377 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.072355 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 01:16:45.072480 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.072409 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 01:16:45.072480 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.072367 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 01:16:45.145133 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.145104 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:45.192184 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.192142 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:45.227390 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.227366 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:45.227523 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.227402 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf6f5\" (UniqueName: \"kubernetes.io/projected/0974aa23-1b99-46bd-a1b0-fdc7f7657069-kube-api-access-bf6f5\") pod \"ingress-canary-zczs4\" (UID: \"0974aa23-1b99-46bd-a1b0-fdc7f7657069\") " pod="openshift-ingress-canary/ingress-canary-zczs4" Sep 16 01:16:45.227523 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.227498 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36024d3d-ea39-4326-b503-4d8a07691efa-tmp-dir\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:45.227640 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.227531 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7srf7\" (UniqueName: \"kubernetes.io/projected/36024d3d-ea39-4326-b503-4d8a07691efa-kube-api-access-7srf7\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:45.227640 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.227566 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36024d3d-ea39-4326-b503-4d8a07691efa-config-volume\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:45.227723 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.227637 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0974aa23-1b99-46bd-a1b0-fdc7f7657069-cert\") pod \"ingress-canary-zczs4\" (UID: \"0974aa23-1b99-46bd-a1b0-fdc7f7657069\") " pod="openshift-ingress-canary/ingress-canary-zczs4" Sep 16 01:16:45.289229 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.289209 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:16:45.289326 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.289238 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:16:45.291609 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.291591 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 01:16:45.292255 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.292238 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 01:16:45.292344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.292330 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zwsmz\"" Sep 16 01:16:45.292402 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.292332 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w6h4t\"" Sep 16 01:16:45.292402 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.292335 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 01:16:45.328567 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.328511 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf6f5\" (UniqueName: \"kubernetes.io/projected/0974aa23-1b99-46bd-a1b0-fdc7f7657069-kube-api-access-bf6f5\") pod \"ingress-canary-zczs4\" (UID: \"0974aa23-1b99-46bd-a1b0-fdc7f7657069\") " pod="openshift-ingress-canary/ingress-canary-zczs4" Sep 16 01:16:45.328672 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.328579 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36024d3d-ea39-4326-b503-4d8a07691efa-tmp-dir\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:45.328672 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.328605 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7srf7\" (UniqueName: \"kubernetes.io/projected/36024d3d-ea39-4326-b503-4d8a07691efa-kube-api-access-7srf7\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:45.328672 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.328636 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36024d3d-ea39-4326-b503-4d8a07691efa-config-volume\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:45.328799 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.328694 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0974aa23-1b99-46bd-a1b0-fdc7f7657069-cert\") pod \"ingress-canary-zczs4\" (UID: \"0974aa23-1b99-46bd-a1b0-fdc7f7657069\") " pod="openshift-ingress-canary/ingress-canary-zczs4" Sep 16 01:16:45.328799 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.328724 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:45.328960 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:45.328835 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 01:16:45.328960 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:45.328888 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls podName:36024d3d-ea39-4326-b503-4d8a07691efa nodeName:}" failed. No retries permitted until 2026-09-16 01:16:45.828870177 +0000 UTC m=+36.081292766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls") pod "dns-default-lnb9f" (UID: "36024d3d-ea39-4326-b503-4d8a07691efa") : secret "dns-default-metrics-tls" not found Sep 16 01:16:45.328960 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.328904 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36024d3d-ea39-4326-b503-4d8a07691efa-tmp-dir\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:45.329251 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.329233 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36024d3d-ea39-4326-b503-4d8a07691efa-config-volume\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:45.333676 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.333647 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0974aa23-1b99-46bd-a1b0-fdc7f7657069-cert\") pod \"ingress-canary-zczs4\" (UID: \"0974aa23-1b99-46bd-a1b0-fdc7f7657069\") " pod="openshift-ingress-canary/ingress-canary-zczs4" Sep 16 01:16:45.337890 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.337873 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf6f5\" (UniqueName: \"kubernetes.io/projected/0974aa23-1b99-46bd-a1b0-fdc7f7657069-kube-api-access-bf6f5\") pod \"ingress-canary-zczs4\" (UID: \"0974aa23-1b99-46bd-a1b0-fdc7f7657069\") " pod="openshift-ingress-canary/ingress-canary-zczs4" Sep 16 01:16:45.338037 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.338021 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7srf7\" (UniqueName: \"kubernetes.io/projected/36024d3d-ea39-4326-b503-4d8a07691efa-kube-api-access-7srf7\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:45.380121 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.380092 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zczs4" Sep 16 01:16:45.699592 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.699554 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zczs4"] Sep 16 01:16:45.704102 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:45.704074 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0974aa23_1b99_46bd_a1b0_fdc7f7657069.slice/crio-3d5f6e97aee224df27c941572b5ab777b30bf96bb9694e1383853a538f1e3dfa WatchSource:0}: Error finding container 3d5f6e97aee224df27c941572b5ab777b30bf96bb9694e1383853a538f1e3dfa: Status 404 returned error can't find the container with id 3d5f6e97aee224df27c941572b5ab777b30bf96bb9694e1383853a538f1e3dfa Sep 16 01:16:45.831862 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:45.831831 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:45.832017 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:45.831951 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 01:16:45.832072 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:45.832018 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls podName:36024d3d-ea39-4326-b503-4d8a07691efa nodeName:}" failed. No retries permitted until 2026-09-16 01:16:46.831998597 +0000 UTC m=+37.084421169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls") pod "dns-default-lnb9f" (UID: "36024d3d-ea39-4326-b503-4d8a07691efa") : secret "dns-default-metrics-tls" not found Sep 16 01:16:46.293332 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:46.293297 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:16:46.296064 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:46.296042 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 01:16:46.556904 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:46.556815 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zczs4" event={"ID":"0974aa23-1b99-46bd-a1b0-fdc7f7657069","Type":"ContainerStarted","Data":"3d5f6e97aee224df27c941572b5ab777b30bf96bb9694e1383853a538f1e3dfa"} Sep 16 01:16:46.840509 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:46.840430 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:46.840676 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:46.840600 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 01:16:46.840676 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:46.840659 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls podName:36024d3d-ea39-4326-b503-4d8a07691efa nodeName:}" failed. No retries permitted until 2026-09-16 01:16:48.840641608 +0000 UTC m=+39.093064175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls") pod "dns-default-lnb9f" (UID: "36024d3d-ea39-4326-b503-4d8a07691efa") : secret "dns-default-metrics-tls" not found Sep 16 01:16:48.855348 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:48.855308 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:48.855848 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:48.855429 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 01:16:48.855848 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:48.855494 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls podName:36024d3d-ea39-4326-b503-4d8a07691efa nodeName:}" failed. No retries permitted until 2026-09-16 01:16:52.855478115 +0000 UTC m=+43.107900682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls") pod "dns-default-lnb9f" (UID: "36024d3d-ea39-4326-b503-4d8a07691efa") : secret "dns-default-metrics-tls" not found Sep 16 01:16:49.326290 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.326254 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:49.378290 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.378262 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:49.516828 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.516776 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-2gmqp"] Sep 16 01:16:49.525033 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.525005 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.527762 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.527686 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 16 01:16:49.528536 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.527943 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 16 01:16:49.528536 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.528346 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 16 01:16:49.528688 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.528582 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-2gmqp"] Sep 16 01:16:49.530027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.529389 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-72tgs\"" Sep 16 01:16:49.530027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.529825 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 16 01:16:49.533596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.533409 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 16 01:16:49.563651 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.563631 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:49.621810 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.621785 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f9887d665-d8xxn"] Sep 16 01:16:49.640574 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.640550 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f9887d665-d8xxn"] Sep 16 01:16:49.640726 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.640657 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.642818 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.642799 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 01:16:49.643009 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.642994 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-79qw7\"" Sep 16 01:16:49.643313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.643300 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 01:16:49.643357 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.643340 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 01:16:49.647095 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.647059 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:49.651923 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.651906 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 01:16:49.660120 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.660102 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d8685e8-58d2-439a-9879-37eeb804efa2-trusted-ca\") pod \"console-operator-696884fb9c-2gmqp\" (UID: \"9d8685e8-58d2-439a-9879-37eeb804efa2\") " pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.660231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.660145 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d8685e8-58d2-439a-9879-37eeb804efa2-config\") pod \"console-operator-696884fb9c-2gmqp\" (UID: \"9d8685e8-58d2-439a-9879-37eeb804efa2\") " pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.660231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.660199 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8685e8-58d2-439a-9879-37eeb804efa2-serving-cert\") pod \"console-operator-696884fb9c-2gmqp\" (UID: \"9d8685e8-58d2-439a-9879-37eeb804efa2\") " pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.660231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.660222 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjspp\" (UniqueName: \"kubernetes.io/projected/9d8685e8-58d2-439a-9879-37eeb804efa2-kube-api-access-mjspp\") pod \"console-operator-696884fb9c-2gmqp\" (UID: \"9d8685e8-58d2-439a-9879-37eeb804efa2\") " pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.715358 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.715304 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-dgbrm"] Sep 16 01:16:49.727093 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.727064 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-dgbrm"] Sep 16 01:16:49.727093 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.727092 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.731201 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.731181 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 16 01:16:49.731392 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.731316 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8h4vn\"" Sep 16 01:16:49.731392 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.731186 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 16 01:16:49.731761 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.731646 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 01:16:49.732596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.732579 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 01:16:49.760914 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.760883 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 16 01:16:49.761117 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.761093 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4be6a433-aedd-4d7a-98da-58077fbed697-trusted-ca\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.761195 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.761141 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn9mq\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-kube-api-access-fn9mq\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.761240 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.761215 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d8685e8-58d2-439a-9879-37eeb804efa2-trusted-ca\") pod \"console-operator-696884fb9c-2gmqp\" (UID: \"9d8685e8-58d2-439a-9879-37eeb804efa2\") " pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.761286 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.761243 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-bound-sa-token\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.761286 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.761270 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4be6a433-aedd-4d7a-98da-58077fbed697-installation-pull-secrets\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.761371 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.761319 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d8685e8-58d2-439a-9879-37eeb804efa2-config\") pod \"console-operator-696884fb9c-2gmqp\" (UID: \"9d8685e8-58d2-439a-9879-37eeb804efa2\") " pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.761371 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.761355 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.761445 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.761402 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4be6a433-aedd-4d7a-98da-58077fbed697-registry-certificates\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.761559 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.761544 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8685e8-58d2-439a-9879-37eeb804efa2-serving-cert\") pod \"console-operator-696884fb9c-2gmqp\" (UID: \"9d8685e8-58d2-439a-9879-37eeb804efa2\") " pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.761604 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.761586 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjspp\" (UniqueName: \"kubernetes.io/projected/9d8685e8-58d2-439a-9879-37eeb804efa2-kube-api-access-mjspp\") pod \"console-operator-696884fb9c-2gmqp\" (UID: \"9d8685e8-58d2-439a-9879-37eeb804efa2\") " pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.761645 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.761623 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4be6a433-aedd-4d7a-98da-58077fbed697-image-registry-private-configuration\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.761684 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.761674 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4be6a433-aedd-4d7a-98da-58077fbed697-ca-trust-extracted\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.762708 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.762681 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d8685e8-58d2-439a-9879-37eeb804efa2-trusted-ca\") pod \"console-operator-696884fb9c-2gmqp\" (UID: \"9d8685e8-58d2-439a-9879-37eeb804efa2\") " pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.765384 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.765360 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8685e8-58d2-439a-9879-37eeb804efa2-serving-cert\") pod \"console-operator-696884fb9c-2gmqp\" (UID: \"9d8685e8-58d2-439a-9879-37eeb804efa2\") " pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.773112 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.773089 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d8685e8-58d2-439a-9879-37eeb804efa2-config\") pod \"console-operator-696884fb9c-2gmqp\" (UID: \"9d8685e8-58d2-439a-9879-37eeb804efa2\") " pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.783913 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.783890 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjspp\" (UniqueName: \"kubernetes.io/projected/9d8685e8-58d2-439a-9879-37eeb804efa2-kube-api-access-mjspp\") pod \"console-operator-696884fb9c-2gmqp\" (UID: \"9d8685e8-58d2-439a-9879-37eeb804efa2\") " pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.794101 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.794076 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:49.836966 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.836940 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:49.862443 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862414 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.862737 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862459 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4be6a433-aedd-4d7a-98da-58077fbed697-registry-certificates\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.862737 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862494 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d157e624-20f1-4d08-a7aa-23a3b0b1d190-serving-cert\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.862737 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862538 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d157e624-20f1-4d08-a7aa-23a3b0b1d190-snapshots\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.862737 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:49.862570 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 01:16:49.862737 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:49.862591 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f9887d665-d8xxn: secret "image-registry-tls" not found Sep 16 01:16:49.862737 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862576 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d157e624-20f1-4d08-a7aa-23a3b0b1d190-tmp\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.862737 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:49.862648 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls podName:4be6a433-aedd-4d7a-98da-58077fbed697 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:50.362625302 +0000 UTC m=+40.615047875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls") pod "image-registry-f9887d665-d8xxn" (UID: "4be6a433-aedd-4d7a-98da-58077fbed697") : secret "image-registry-tls" not found Sep 16 01:16:49.862737 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862685 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4be6a433-aedd-4d7a-98da-58077fbed697-image-registry-private-configuration\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.862737 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862712 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl54d\" (UniqueName: \"kubernetes.io/projected/d157e624-20f1-4d08-a7aa-23a3b0b1d190-kube-api-access-kl54d\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.863176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862747 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4be6a433-aedd-4d7a-98da-58077fbed697-ca-trust-extracted\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.863176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862765 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4be6a433-aedd-4d7a-98da-58077fbed697-trusted-ca\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.863176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862825 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn9mq\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-kube-api-access-fn9mq\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.863176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862891 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-bound-sa-token\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.863176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862922 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d157e624-20f1-4d08-a7aa-23a3b0b1d190-service-ca-bundle\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.863176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862954 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4be6a433-aedd-4d7a-98da-58077fbed697-installation-pull-secrets\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.863176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.862978 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d157e624-20f1-4d08-a7aa-23a3b0b1d190-trusted-ca-bundle\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.863176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.863044 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4be6a433-aedd-4d7a-98da-58077fbed697-registry-certificates\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.863176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.863139 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4be6a433-aedd-4d7a-98da-58077fbed697-ca-trust-extracted\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.863580 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.863562 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4be6a433-aedd-4d7a-98da-58077fbed697-trusted-ca\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.865748 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.865726 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4be6a433-aedd-4d7a-98da-58077fbed697-installation-pull-secrets\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.865824 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.865766 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4be6a433-aedd-4d7a-98da-58077fbed697-image-registry-private-configuration\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.870781 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.870751 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-bound-sa-token\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.870951 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.870930 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn9mq\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-kube-api-access-fn9mq\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:49.895097 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.894741 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:49.963756 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.963716 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d157e624-20f1-4d08-a7aa-23a3b0b1d190-serving-cert\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.963896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.963773 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d157e624-20f1-4d08-a7aa-23a3b0b1d190-snapshots\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.963896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.963810 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d157e624-20f1-4d08-a7aa-23a3b0b1d190-tmp\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.963896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.963841 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl54d\" (UniqueName: \"kubernetes.io/projected/d157e624-20f1-4d08-a7aa-23a3b0b1d190-kube-api-access-kl54d\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.964051 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.963922 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d157e624-20f1-4d08-a7aa-23a3b0b1d190-service-ca-bundle\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.964051 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.963952 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d157e624-20f1-4d08-a7aa-23a3b0b1d190-trusted-ca-bundle\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.964549 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.964524 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d157e624-20f1-4d08-a7aa-23a3b0b1d190-snapshots\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.964665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.964528 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d157e624-20f1-4d08-a7aa-23a3b0b1d190-service-ca-bundle\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.964665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.964626 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d157e624-20f1-4d08-a7aa-23a3b0b1d190-tmp\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.964810 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.964783 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d157e624-20f1-4d08-a7aa-23a3b0b1d190-trusted-ca-bundle\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.966037 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.966016 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d157e624-20f1-4d08-a7aa-23a3b0b1d190-serving-cert\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:49.976095 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:49.976073 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl54d\" (UniqueName: \"kubernetes.io/projected/d157e624-20f1-4d08-a7aa-23a3b0b1d190-kube-api-access-kl54d\") pod \"insights-operator-5844b856db-dgbrm\" (UID: \"d157e624-20f1-4d08-a7aa-23a3b0b1d190\") " pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:50.037071 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:50.037052 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-dgbrm" Sep 16 01:16:50.114866 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:50.114660 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-2gmqp"] Sep 16 01:16:50.119907 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:50.119868 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8685e8_58d2_439a_9879_37eeb804efa2.slice/crio-a1d26af32d9fee97d8f42c5156e5d29d5d82aa5a893dc0abc613c5f8affe4ff5 WatchSource:0}: Error finding container a1d26af32d9fee97d8f42c5156e5d29d5d82aa5a893dc0abc613c5f8affe4ff5: Status 404 returned error can't find the container with id a1d26af32d9fee97d8f42c5156e5d29d5d82aa5a893dc0abc613c5f8affe4ff5 Sep 16 01:16:50.339016 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:50.338986 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-dgbrm"] Sep 16 01:16:50.341699 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:50.341678 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd157e624_20f1_4d08_a7aa_23a3b0b1d190.slice/crio-39183db71fd7e1a48e2bd13c44ee3a1d6a4196015498bd6ea9a2252596d1dcba WatchSource:0}: Error finding container 39183db71fd7e1a48e2bd13c44ee3a1d6a4196015498bd6ea9a2252596d1dcba: Status 404 returned error can't find the container with id 39183db71fd7e1a48e2bd13c44ee3a1d6a4196015498bd6ea9a2252596d1dcba Sep 16 01:16:50.366514 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:50.366455 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:50.366619 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:50.366604 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 01:16:50.366659 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:50.366622 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f9887d665-d8xxn: secret "image-registry-tls" not found Sep 16 01:16:50.366694 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:50.366668 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls podName:4be6a433-aedd-4d7a-98da-58077fbed697 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:51.3666542 +0000 UTC m=+41.619076767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls") pod "image-registry-f9887d665-d8xxn" (UID: "4be6a433-aedd-4d7a-98da-58077fbed697") : secret "image-registry-tls" not found Sep 16 01:16:50.565684 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:50.565653 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dgbrm" event={"ID":"d157e624-20f1-4d08-a7aa-23a3b0b1d190","Type":"ContainerStarted","Data":"39183db71fd7e1a48e2bd13c44ee3a1d6a4196015498bd6ea9a2252596d1dcba"} Sep 16 01:16:50.566859 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:50.566838 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" event={"ID":"9d8685e8-58d2-439a-9879-37eeb804efa2","Type":"ContainerStarted","Data":"a1d26af32d9fee97d8f42c5156e5d29d5d82aa5a893dc0abc613c5f8affe4ff5"} Sep 16 01:16:50.569610 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:50.569589 2558 generic.go:356] "Generic (PLEG): container finished" podID="0b2afed4-62a6-41d3-b80b-61e37294be11" containerID="cf04b79dbc9a194e9b291c6272ff0806c11070f65d138bb4e9050a7b4bbb0a97" exitCode=0 Sep 16 01:16:50.569705 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:50.569647 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjqv9" event={"ID":"0b2afed4-62a6-41d3-b80b-61e37294be11","Type":"ContainerDied","Data":"cf04b79dbc9a194e9b291c6272ff0806c11070f65d138bb4e9050a7b4bbb0a97"} Sep 16 01:16:50.571226 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:50.571207 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zczs4" event={"ID":"0974aa23-1b99-46bd-a1b0-fdc7f7657069","Type":"ContainerStarted","Data":"ed63b396882bc17c44ba8136958effcdb373d2c03b57dc050c23d15108724507"} Sep 16 01:16:50.601746 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:50.601709 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zczs4" podStartSLOduration=1.7519000070000001 podStartE2EDuration="5.60169622s" totalImagesPullingTime="3.849796213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:50.601411215 +0000 UTC m=+40.853833803" watchObservedRunningTime="2026-09-16 01:16:50.60169622 +0000 UTC m=+40.854118811" Sep 16 01:16:51.375367 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:51.375320 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:51.375888 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:51.375490 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 01:16:51.375888 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:51.375513 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f9887d665-d8xxn: secret "image-registry-tls" not found Sep 16 01:16:51.375888 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:51.375585 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls podName:4be6a433-aedd-4d7a-98da-58077fbed697 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:53.37556301 +0000 UTC m=+43.627985600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls") pod "image-registry-f9887d665-d8xxn" (UID: "4be6a433-aedd-4d7a-98da-58077fbed697") : secret "image-registry-tls" not found Sep 16 01:16:51.562842 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:51.562798 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:51.577353 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:51.577314 2558 generic.go:356] "Generic (PLEG): container finished" podID="0b2afed4-62a6-41d3-b80b-61e37294be11" containerID="e2e07b04156915cfe6d261a45ddfa700b49e58b5b3b9ba97ed0a58e29d456164" exitCode=0 Sep 16 01:16:51.577544 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:51.577371 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjqv9" event={"ID":"0b2afed4-62a6-41d3-b80b-61e37294be11","Type":"ContainerDied","Data":"e2e07b04156915cfe6d261a45ddfa700b49e58b5b3b9ba97ed0a58e29d456164"} Sep 16 01:16:51.630879 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:51.630806 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:51.684050 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:51.684004 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:52.888215 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:52.888188 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:16:52.888507 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:52.888332 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 01:16:52.888507 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:52.888391 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls podName:36024d3d-ea39-4326-b503-4d8a07691efa nodeName:}" failed. No retries permitted until 2026-09-16 01:17:00.88837549 +0000 UTC m=+51.140798057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls") pod "dns-default-lnb9f" (UID: "36024d3d-ea39-4326-b503-4d8a07691efa") : secret "dns-default-metrics-tls" not found Sep 16 01:16:53.392578 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:53.392544 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:53.392706 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:53.392681 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 01:16:53.392706 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:53.392699 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f9887d665-d8xxn: secret "image-registry-tls" not found Sep 16 01:16:53.392831 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:53.392755 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls podName:4be6a433-aedd-4d7a-98da-58077fbed697 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:57.39273812 +0000 UTC m=+47.645160688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls") pod "image-registry-f9887d665-d8xxn" (UID: "4be6a433-aedd-4d7a-98da-58077fbed697") : secret "image-registry-tls" not found Sep 16 01:16:53.584329 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:53.584304 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/0.log" Sep 16 01:16:53.584432 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:53.584350 2558 generic.go:356] "Generic (PLEG): container finished" podID="9d8685e8-58d2-439a-9879-37eeb804efa2" containerID="1d167ed14d34bea0883d139380470e5ae2be8c82b5019165a2574408c9a10d64" exitCode=255 Sep 16 01:16:53.584498 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:53.584437 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" event={"ID":"9d8685e8-58d2-439a-9879-37eeb804efa2","Type":"ContainerDied","Data":"1d167ed14d34bea0883d139380470e5ae2be8c82b5019165a2574408c9a10d64"} Sep 16 01:16:53.584610 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:53.584595 2558 scope.go:117] "RemoveContainer" containerID="1d167ed14d34bea0883d139380470e5ae2be8c82b5019165a2574408c9a10d64" Sep 16 01:16:53.588355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:53.588334 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjqv9" event={"ID":"0b2afed4-62a6-41d3-b80b-61e37294be11","Type":"ContainerStarted","Data":"220c72bd70391629eb65be83a93372e7bd844a19ca2a75d4ef414a6a78d73262"} Sep 16 01:16:53.589824 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:53.589803 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dgbrm" event={"ID":"d157e624-20f1-4d08-a7aa-23a3b0b1d190","Type":"ContainerStarted","Data":"2a7e8f66015d6d1bc9c2c6492b738bc89e21a9079df5246f1a0f0178087835a2"} Sep 16 01:16:53.613552 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:53.613509 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-dgbrm" podStartSLOduration=1.8552243659999998 podStartE2EDuration="4.613497156s" totalImagesPullingTime="2.75827279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:53.612418296 +0000 UTC m=+43.864840908" watchObservedRunningTime="2026-09-16 01:16:53.613497156 +0000 UTC m=+43.865919745" Sep 16 01:16:53.631123 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:53.631073 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fjqv9" podStartSLOduration=11.211649186 podStartE2EDuration="43.631055727s" totalImagesPullingTime="32.419406541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:16:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:53.629148615 +0000 UTC m=+43.881571205" watchObservedRunningTime="2026-09-16 01:16:53.631055727 +0000 UTC m=+43.883478317" Sep 16 01:16:53.750604 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:53.750578 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:53.847834 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:53.847801 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:54.224588 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.224558 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rrxxj"] Sep 16 01:16:54.245129 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.245105 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rrxxj"] Sep 16 01:16:54.245299 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.245255 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.247436 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.247412 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 01:16:54.247603 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.247469 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ncdjs\"" Sep 16 01:16:54.247808 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.247760 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 01:16:54.266660 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.266630 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:54.323142 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.323107 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:54.399168 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.399126 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.399283 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.399195 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9633a4e7-63e6-44e1-9469-7334c5dc56e4-crio-socket\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.399351 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.399277 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9633a4e7-63e6-44e1-9469-7334c5dc56e4-data-volume\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.399351 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.399307 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg69d\" (UniqueName: \"kubernetes.io/projected/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-api-access-cg69d\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.399417 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.399375 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.499655 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.499627 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9633a4e7-63e6-44e1-9469-7334c5dc56e4-data-volume\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.499755 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.499661 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg69d\" (UniqueName: \"kubernetes.io/projected/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-api-access-cg69d\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.499755 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.499691 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.499755 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.499745 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.499904 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.499774 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9633a4e7-63e6-44e1-9469-7334c5dc56e4-crio-socket\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.499904 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:54.499801 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 01:16:54.499904 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:54.499827 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 01:16:54.499904 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:54.499882 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls podName:9633a4e7-63e6-44e1-9469-7334c5dc56e4 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:54.999864075 +0000 UTC m=+45.252286656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rrxxj" (UID: "9633a4e7-63e6-44e1-9469-7334c5dc56e4") : secret "insights-runtime-extractor-tls" not found Sep 16 01:16:54.499904 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:54.499905 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm podName:9633a4e7-63e6-44e1-9469-7334c5dc56e4 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:54.999895833 +0000 UTC m=+45.252318401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rrxxj" (UID: "9633a4e7-63e6-44e1-9469-7334c5dc56e4") : configmap "kube-rbac-proxy" not found Sep 16 01:16:54.500123 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.499932 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9633a4e7-63e6-44e1-9469-7334c5dc56e4-crio-socket\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.500123 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.499980 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9633a4e7-63e6-44e1-9469-7334c5dc56e4-data-volume\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.507048 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.507023 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg69d\" (UniqueName: \"kubernetes.io/projected/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-api-access-cg69d\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:54.593848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.593825 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:16:54.594267 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.594253 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/0.log" Sep 16 01:16:54.594330 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.594292 2558 generic.go:356] "Generic (PLEG): container finished" podID="9d8685e8-58d2-439a-9879-37eeb804efa2" containerID="e992aa6a9141a24776032e881ef1e8bbcf98d4f05306a4d10ac3c877dffd6592" exitCode=255 Sep 16 01:16:54.594398 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.594376 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" event={"ID":"9d8685e8-58d2-439a-9879-37eeb804efa2","Type":"ContainerDied","Data":"e992aa6a9141a24776032e881ef1e8bbcf98d4f05306a4d10ac3c877dffd6592"} Sep 16 01:16:54.594448 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.594428 2558 scope.go:117] "RemoveContainer" containerID="1d167ed14d34bea0883d139380470e5ae2be8c82b5019165a2574408c9a10d64" Sep 16 01:16:54.594608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.594589 2558 scope.go:117] "RemoveContainer" containerID="e992aa6a9141a24776032e881ef1e8bbcf98d4f05306a4d10ac3c877dffd6592" Sep 16 01:16:54.594837 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:54.594810 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-2gmqp_openshift-console-operator(9d8685e8-58d2-439a-9879-37eeb804efa2)\"" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" podUID="9d8685e8-58d2-439a-9879-37eeb804efa2" Sep 16 01:16:54.600486 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.600461 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:54.653422 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:54.653370 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:55.002991 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.002926 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:55.003108 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.002995 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:55.003108 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:55.003059 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 01:16:55.003108 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:55.003078 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 01:16:55.003263 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:55.003114 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm podName:9633a4e7-63e6-44e1-9469-7334c5dc56e4 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:56.003101258 +0000 UTC m=+46.255523825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rrxxj" (UID: "9633a4e7-63e6-44e1-9469-7334c5dc56e4") : configmap "kube-rbac-proxy" not found Sep 16 01:16:55.003263 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:55.003137 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls podName:9633a4e7-63e6-44e1-9469-7334c5dc56e4 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:56.003120477 +0000 UTC m=+46.255543063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rrxxj" (UID: "9633a4e7-63e6-44e1-9469-7334c5dc56e4") : secret "insights-runtime-extractor-tls" not found Sep 16 01:16:55.025403 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.025373 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:55.078229 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.078207 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:55.300304 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.300242 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:55.400072 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.400049 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:55.574116 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.574056 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:55.589437 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.589408 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-gwbth"] Sep 16 01:16:55.598010 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.597993 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:16:55.598272 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.598261 2558 scope.go:117] "RemoveContainer" containerID="e992aa6a9141a24776032e881ef1e8bbcf98d4f05306a4d10ac3c877dffd6592" Sep 16 01:16:55.598411 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:55.598394 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-2gmqp_openshift-console-operator(9d8685e8-58d2-439a-9879-37eeb804efa2)\"" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" podUID="9d8685e8-58d2-439a-9879-37eeb804efa2" Sep 16 01:16:55.620567 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.620543 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-gwbth"] Sep 16 01:16:55.620656 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.620633 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-gwbth" Sep 16 01:16:55.622593 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.622571 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 16 01:16:55.622686 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.622591 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 16 01:16:55.622686 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.622626 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qq75h\"" Sep 16 01:16:55.636633 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.636610 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:55.698510 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.698491 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:55.809286 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.809261 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqrkw\" (UniqueName: \"kubernetes.io/projected/6f156ed2-e2d7-4711-abb7-91765eca5f53-kube-api-access-fqrkw\") pod \"migrator-8b4d697c5-gwbth\" (UID: \"6f156ed2-e2d7-4711-abb7-91765eca5f53\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-gwbth" Sep 16 01:16:55.909822 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.909798 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqrkw\" (UniqueName: \"kubernetes.io/projected/6f156ed2-e2d7-4711-abb7-91765eca5f53-kube-api-access-fqrkw\") pod \"migrator-8b4d697c5-gwbth\" (UID: \"6f156ed2-e2d7-4711-abb7-91765eca5f53\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-gwbth" Sep 16 01:16:55.917039 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.917021 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqrkw\" (UniqueName: \"kubernetes.io/projected/6f156ed2-e2d7-4711-abb7-91765eca5f53-kube-api-access-fqrkw\") pod \"migrator-8b4d697c5-gwbth\" (UID: \"6f156ed2-e2d7-4711-abb7-91765eca5f53\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-gwbth" Sep 16 01:16:55.929047 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:55.929031 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-gwbth" Sep 16 01:16:56.010654 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:56.010625 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:56.010794 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:56.010683 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:56.010860 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:56.010802 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 01:16:56.010860 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:56.010848 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls podName:9633a4e7-63e6-44e1-9469-7334c5dc56e4 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:58.010835769 +0000 UTC m=+48.263258335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rrxxj" (UID: "9633a4e7-63e6-44e1-9469-7334c5dc56e4") : secret "insights-runtime-extractor-tls" not found Sep 16 01:16:56.011169 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:56.011139 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 01:16:56.011237 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:56.011206 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm podName:9633a4e7-63e6-44e1-9469-7334c5dc56e4 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:58.011193026 +0000 UTC m=+48.263615593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rrxxj" (UID: "9633a4e7-63e6-44e1-9469-7334c5dc56e4") : configmap "kube-rbac-proxy" not found Sep 16 01:16:56.184330 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:56.184272 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9twcf_f6be6ffd-8560-4250-84a4-0741806d4858/dns-node-resolver/0.log" Sep 16 01:16:56.217883 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:56.217856 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-gwbth"] Sep 16 01:16:56.220663 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:16:56.220639 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f156ed2_e2d7_4711_abb7_91765eca5f53.slice/crio-9ee2150b0a9a71b98893a5ff97f5b3e71acd39e9a4382405d0410e2e78e7332a WatchSource:0}: Error finding container 9ee2150b0a9a71b98893a5ff97f5b3e71acd39e9a4382405d0410e2e78e7332a: Status 404 returned error can't find the container with id 9ee2150b0a9a71b98893a5ff97f5b3e71acd39e9a4382405d0410e2e78e7332a Sep 16 01:16:56.601652 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:56.601572 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-gwbth" event={"ID":"6f156ed2-e2d7-4711-abb7-91765eca5f53","Type":"ContainerStarted","Data":"9ee2150b0a9a71b98893a5ff97f5b3e71acd39e9a4382405d0410e2e78e7332a"} Sep 16 01:16:56.935969 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:56.935932 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:56.983793 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:56.983770 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8s8qx_2a63a0c6-8521-4839-9955-1d4f7e6ea1a9/node-ca/0.log" Sep 16 01:16:57.421598 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:57.421566 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:16:57.421783 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:57.421727 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 01:16:57.421783 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:57.421748 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f9887d665-d8xxn: secret "image-registry-tls" not found Sep 16 01:16:57.421852 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:57.421816 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls podName:4be6a433-aedd-4d7a-98da-58077fbed697 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:05.421795926 +0000 UTC m=+55.674218496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls") pod "image-registry-f9887d665-d8xxn" (UID: "4be6a433-aedd-4d7a-98da-58077fbed697") : secret "image-registry-tls" not found Sep 16 01:16:57.806777 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:57.806668 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:58.026479 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:58.026450 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:58.026590 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:58.026549 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:16:58.026590 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:58.026568 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 01:16:58.026682 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:58.026632 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm podName:9633a4e7-63e6-44e1-9469-7334c5dc56e4 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:02.02661501 +0000 UTC m=+52.279037577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rrxxj" (UID: "9633a4e7-63e6-44e1-9469-7334c5dc56e4") : configmap "kube-rbac-proxy" not found Sep 16 01:16:58.026682 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:58.026636 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 01:16:58.026682 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:58.026680 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls podName:9633a4e7-63e6-44e1-9469-7334c5dc56e4 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:02.026668208 +0000 UTC m=+52.279090775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rrxxj" (UID: "9633a4e7-63e6-44e1-9469-7334c5dc56e4") : secret "insights-runtime-extractor-tls" not found Sep 16 01:16:58.183707 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:58.183684 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zczs4_0974aa23-1b99-46bd-a1b0-fdc7f7657069/serve-healthcheck-canary/0.log" Sep 16 01:16:58.607490 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:58.607457 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-gwbth" event={"ID":"6f156ed2-e2d7-4711-abb7-91765eca5f53","Type":"ContainerStarted","Data":"d9e367dfb122f708d83fcef48b561683ff1168561efdea9285a81bfda11bac0b"} Sep 16 01:16:58.607658 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:58.607506 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-gwbth" event={"ID":"6f156ed2-e2d7-4711-abb7-91765eca5f53","Type":"ContainerStarted","Data":"fd14d43c411586bdbc833d3690c8c74aec0b3835fd3862e4daf9c9315adc740e"} Sep 16 01:16:58.624767 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:58.624725 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-gwbth" podStartSLOduration=1.94391778 podStartE2EDuration="3.62471162s" totalImagesPullingTime="1.68079384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:58.623993187 +0000 UTC m=+48.876415776" watchObservedRunningTime="2026-09-16 01:16:58.62471162 +0000 UTC m=+48.877134262" Sep 16 01:16:59.837551 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:59.837522 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:59.837955 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:59.837596 2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:16:59.837955 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:16:59.837863 2558 scope.go:117] "RemoveContainer" containerID="e992aa6a9141a24776032e881ef1e8bbcf98d4f05306a4d10ac3c877dffd6592" Sep 16 01:16:59.838024 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:16:59.837999 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-2gmqp_openshift-console-operator(9d8685e8-58d2-439a-9879-37eeb804efa2)\"" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" podUID="9d8685e8-58d2-439a-9879-37eeb804efa2" Sep 16 01:17:00.042067 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:00.042036 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:00.946605 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:00.946573 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:17:00.946938 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:00.946707 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 01:17:00.946938 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:00.946770 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls podName:36024d3d-ea39-4326-b503-4d8a07691efa nodeName:}" failed. No retries permitted until 2026-09-16 01:17:16.946754927 +0000 UTC m=+67.199177497 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls") pod "dns-default-lnb9f" (UID: "36024d3d-ea39-4326-b503-4d8a07691efa") : secret "dns-default-metrics-tls" not found Sep 16 01:17:02.053937 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:02.053902 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:17:02.054328 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:02.053973 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:17:02.054328 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:02.054038 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 01:17:02.054328 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:02.054058 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 01:17:02.054328 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:02.054098 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls podName:9633a4e7-63e6-44e1-9469-7334c5dc56e4 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:10.054083509 +0000 UTC m=+60.306506076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rrxxj" (UID: "9633a4e7-63e6-44e1-9469-7334c5dc56e4") : secret "insights-runtime-extractor-tls" not found Sep 16 01:17:02.054328 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:02.054112 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm podName:9633a4e7-63e6-44e1-9469-7334c5dc56e4 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:10.054105974 +0000 UTC m=+60.306528541 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rrxxj" (UID: "9633a4e7-63e6-44e1-9469-7334c5dc56e4") : configmap "kube-rbac-proxy" not found Sep 16 01:17:02.617635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:02.617603 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:02.786736 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:02.786710 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:02.888657 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:02.888593 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:03.028172 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:03.028100 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:03.129289 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:03.129251 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:03.193371 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:03.193312 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:03.514309 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:03.514248 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:03.717338 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:03.717308 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:04.296572 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:04.296544 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:04.714275 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:04.714248 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:04.920721 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:04.920695 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:05.312829 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:05.312791 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:05.478629 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:05.478603 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls\") pod \"image-registry-f9887d665-d8xxn\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:17:05.478773 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:05.478757 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 01:17:05.478842 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:05.478777 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f9887d665-d8xxn: secret "image-registry-tls" not found Sep 16 01:17:05.478842 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:05.478837 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls podName:4be6a433-aedd-4d7a-98da-58077fbed697 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:21.478817974 +0000 UTC m=+71.731240544 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls") pod "image-registry-f9887d665-d8xxn" (UID: "4be6a433-aedd-4d7a-98da-58077fbed697") : secret "image-registry-tls" not found Sep 16 01:17:05.513726 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:05.513708 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:05.913393 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:05.913369 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:06.296360 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:06.296302 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:06.396860 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:06.396833 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:07.111034 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:07.111004 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:07.211684 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:07.211658 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:07.381129 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:07.381105 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:07.513613 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:07.513589 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:07.715754 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:07.715685 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:07.944675 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:07.944653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:08.113928 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:08.113909 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:08.809115 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:08.809090 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:09.321726 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:09.321699 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:09.514243 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:09.514216 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:09.545746 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:09.545722 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgv65" Sep 16 01:17:09.712960 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:09.712935 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:09.912423 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:09.912392 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:10.113179 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:10.113101 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:10.115087 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:10.115066 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:17:10.115195 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:10.115179 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:17:10.115261 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:10.115205 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 01:17:10.115300 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:10.115262 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls podName:9633a4e7-63e6-44e1-9469-7334c5dc56e4 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:26.115248983 +0000 UTC m=+76.367671550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rrxxj" (UID: "9633a4e7-63e6-44e1-9469-7334c5dc56e4") : secret "insights-runtime-extractor-tls" not found Sep 16 01:17:10.115622 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:10.115606 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9633a4e7-63e6-44e1-9469-7334c5dc56e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:17:11.116742 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:11.116708 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:11.517184 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:11.517094 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:11.714138 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:11.714103 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:11.917726 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:11.917702 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:12.113221 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:12.113196 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:12.914742 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:12.914706 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:13.118503 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:13.118470 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:13.288912 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:13.288862 2558 scope.go:117] "RemoveContainer" containerID="e992aa6a9141a24776032e881ef1e8bbcf98d4f05306a4d10ac3c877dffd6592" Sep 16 01:17:13.312554 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:13.312531 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:13.432961 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:13.432935 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:13.533291 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:13.533265 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:13.643110 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:13.643089 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:17:13.643247 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:13.643137 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" event={"ID":"9d8685e8-58d2-439a-9879-37eeb804efa2","Type":"ContainerStarted","Data":"2b6fadc8610b5beadb6891f67c1e632e2eae433d80d8375987d0dfbf2d0f4141"} Sep 16 01:17:13.643449 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:13.643428 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:17:13.659495 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:13.659459 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" podStartSLOduration=21.688062235 podStartE2EDuration="24.65944517s" totalImagesPullingTime="2.971382935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:13.658883802 +0000 UTC m=+63.911306390" watchObservedRunningTime="2026-09-16 01:17:13.65944517 +0000 UTC m=+63.911867759" Sep 16 01:17:13.770479 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:13.770458 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-2gmqp" Sep 16 01:17:13.949283 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:13.949198 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:14.051050 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:14.051019 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:14.150910 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:14.150855 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:14.250833 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:14.250775 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:14.622988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:14.622966 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:14.681899 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:14.681863 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:14.824380 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:14.824356 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:14.925201 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:14.925119 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:15.828896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:15.828858 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:15.978112 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:15.978060 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:16.060335 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.060304 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:17:16.060484 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.060341 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjq9j\" (UniqueName: \"kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j\") pod \"network-check-target-q6xx7\" (UID: \"afe22dac-90f9-4893-9e84-0d7731b9fd8f\") " pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:17:16.060484 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.060396 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs\") pod \"network-metrics-daemon-hs6d5\" (UID: \"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c\") " pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:17:16.062438 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.062411 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 01:17:16.062834 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.062817 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 01:17:16.063082 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.063066 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 01:17:16.073133 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.073113 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 01:17:16.073284 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.073266 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c-metrics-certs\") pod \"network-metrics-daemon-hs6d5\" (UID: \"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c\") " pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:17:16.073367 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.073349 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5826c82e-35c3-40b5-a3f8-774437001cf8-original-pull-secret\") pod \"global-pull-secret-syncer-6ndbn\" (UID: \"5826c82e-35c3-40b5-a3f8-774437001cf8\") " pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:17:16.078467 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.078447 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:16.083373 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.083325 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjq9j\" (UniqueName: \"kubernetes.io/projected/afe22dac-90f9-4893-9e84-0d7731b9fd8f-kube-api-access-jjq9j\") pod \"network-check-target-q6xx7\" (UID: \"afe22dac-90f9-4893-9e84-0d7731b9fd8f\") " pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:17:16.201875 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.201856 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w6h4t\"" Sep 16 01:17:16.207361 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.207343 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zwsmz\"" Sep 16 01:17:16.210047 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.210034 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs6d5" Sep 16 01:17:16.216406 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.216382 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:17:16.230258 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.230223 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:16.303998 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.303972 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ndbn" Sep 16 01:17:16.483189 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.483146 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hs6d5"] Sep 16 01:17:16.486309 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.486285 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q6xx7"] Sep 16 01:17:16.486402 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:16.486292 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1e9be_9d70_4ed6_9894_c7b7342a0c0c.slice/crio-8306284465a5597b89ddfd0a00540c9b6cdbeafd13c6174115548d12f22d9249 WatchSource:0}: Error finding container 8306284465a5597b89ddfd0a00540c9b6cdbeafd13c6174115548d12f22d9249: Status 404 returned error can't find the container with id 8306284465a5597b89ddfd0a00540c9b6cdbeafd13c6174115548d12f22d9249 Sep 16 01:17:16.488898 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:16.488880 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe22dac_90f9_4893_9e84_0d7731b9fd8f.slice/crio-f5ba7416045d259e964af09007701f46fadcb0f30fd3a8703361b30ebe7fc08a WatchSource:0}: Error finding container f5ba7416045d259e964af09007701f46fadcb0f30fd3a8703361b30ebe7fc08a: Status 404 returned error can't find the container with id f5ba7416045d259e964af09007701f46fadcb0f30fd3a8703361b30ebe7fc08a Sep 16 01:17:16.568837 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.568782 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6ndbn"] Sep 16 01:17:16.571952 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:16.571928 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5826c82e_35c3_40b5_a3f8_774437001cf8.slice/crio-030cd2fa0e60cef71fb89c44f18ebe4692b1e559c0514d766bb7088016197546 WatchSource:0}: Error finding container 030cd2fa0e60cef71fb89c44f18ebe4692b1e559c0514d766bb7088016197546: Status 404 returned error can't find the container with id 030cd2fa0e60cef71fb89c44f18ebe4692b1e559c0514d766bb7088016197546 Sep 16 01:17:16.655200 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.655176 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6ndbn" event={"ID":"5826c82e-35c3-40b5-a3f8-774437001cf8","Type":"ContainerStarted","Data":"030cd2fa0e60cef71fb89c44f18ebe4692b1e559c0514d766bb7088016197546"} Sep 16 01:17:16.656299 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.656275 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q6xx7" event={"ID":"afe22dac-90f9-4893-9e84-0d7731b9fd8f","Type":"ContainerStarted","Data":"f5ba7416045d259e964af09007701f46fadcb0f30fd3a8703361b30ebe7fc08a"} Sep 16 01:17:16.657338 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.657317 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hs6d5" event={"ID":"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c","Type":"ContainerStarted","Data":"8306284465a5597b89ddfd0a00540c9b6cdbeafd13c6174115548d12f22d9249"} Sep 16 01:17:16.968149 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.968075 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:17:16.970368 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:16.970348 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36024d3d-ea39-4326-b503-4d8a07691efa-metrics-tls\") pod \"dns-default-lnb9f\" (UID: \"36024d3d-ea39-4326-b503-4d8a07691efa\") " pod="openshift-dns/dns-default-lnb9f" Sep 16 01:17:17.168296 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:17.168259 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sq7hb\"" Sep 16 01:17:17.176191 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:17.176153 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnb9f" Sep 16 01:17:17.469670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:17.469643 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lnb9f"] Sep 16 01:17:17.472592 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:17.472563 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36024d3d_ea39_4326_b503_4d8a07691efa.slice/crio-3ee3643818cd2299ad78f7f773312d5487398ff2d40151bbadbd9007df9a071e WatchSource:0}: Error finding container 3ee3643818cd2299ad78f7f773312d5487398ff2d40151bbadbd9007df9a071e: Status 404 returned error can't find the container with id 3ee3643818cd2299ad78f7f773312d5487398ff2d40151bbadbd9007df9a071e Sep 16 01:17:17.664139 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:17.664100 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnb9f" event={"ID":"36024d3d-ea39-4326-b503-4d8a07691efa","Type":"ContainerStarted","Data":"3ee3643818cd2299ad78f7f773312d5487398ff2d40151bbadbd9007df9a071e"} Sep 16 01:17:18.641610 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.641575 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd"] Sep 16 01:17:18.645087 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.645065 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd" Sep 16 01:17:18.649323 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.649296 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 16 01:17:18.649635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.649613 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 16 01:17:18.649843 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.649824 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-6vvvk\"" Sep 16 01:17:18.650047 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.650029 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 16 01:17:18.650286 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.650268 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 16 01:17:18.655399 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.654145 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd"] Sep 16 01:17:18.672694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.672656 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hs6d5" event={"ID":"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c","Type":"ContainerStarted","Data":"555bcf7b6320e89a59a5f5eab6c9609cf49c29119a5a8d5aa7b69f7400fe3704"} Sep 16 01:17:18.672852 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.672724 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hs6d5" event={"ID":"2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c","Type":"ContainerStarted","Data":"685110cd709929432369bf75b1b53668b81e13b6606ff2ea5000579747544ef8"} Sep 16 01:17:18.683111 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.683072 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2klg\" (UniqueName: \"kubernetes.io/projected/2a4b0c50-3ab7-4d56-b612-0a2fcad22161-kube-api-access-j2klg\") pod \"managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd\" (UID: \"2a4b0c50-3ab7-4d56-b612-0a2fcad22161\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd" Sep 16 01:17:18.683258 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.683116 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4b0c50-3ab7-4d56-b612-0a2fcad22161-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd\" (UID: \"2a4b0c50-3ab7-4d56-b612-0a2fcad22161\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd" Sep 16 01:17:18.694711 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.694658 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hs6d5" podStartSLOduration=67.297594056 podStartE2EDuration="1m8.694638158s" totalImagesPullingTime="1.397044102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:18.694386767 +0000 UTC m=+68.946809363" watchObservedRunningTime="2026-09-16 01:17:18.694638158 +0000 UTC m=+68.947060747" Sep 16 01:17:18.723655 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.723625 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f9887d665-d8xxn"] Sep 16 01:17:18.723893 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:18.723871 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-f9887d665-d8xxn" podUID="4be6a433-aedd-4d7a-98da-58077fbed697" Sep 16 01:17:18.749912 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.749820 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:18.765413 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.765387 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-659c7b6595-x87bp"] Sep 16 01:17:18.768464 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.768443 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.785561 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.784506 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2klg\" (UniqueName: \"kubernetes.io/projected/2a4b0c50-3ab7-4d56-b612-0a2fcad22161-kube-api-access-j2klg\") pod \"managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd\" (UID: \"2a4b0c50-3ab7-4d56-b612-0a2fcad22161\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd" Sep 16 01:17:18.785561 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.784549 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4b0c50-3ab7-4d56-b612-0a2fcad22161-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd\" (UID: \"2a4b0c50-3ab7-4d56-b612-0a2fcad22161\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd" Sep 16 01:17:18.785561 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.785451 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-659c7b6595-x87bp"] Sep 16 01:17:18.788756 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.788731 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4b0c50-3ab7-4d56-b612-0a2fcad22161-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd\" (UID: \"2a4b0c50-3ab7-4d56-b612-0a2fcad22161\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd" Sep 16 01:17:18.802570 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.802545 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2klg\" (UniqueName: \"kubernetes.io/projected/2a4b0c50-3ab7-4d56-b612-0a2fcad22161-kube-api-access-j2klg\") pod \"managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd\" (UID: \"2a4b0c50-3ab7-4d56-b612-0a2fcad22161\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd" Sep 16 01:17:18.841482 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.841441 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:18.886119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.885407 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04040d3e-3b48-47f2-8137-55ffd432390d-bound-sa-token\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.886119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.885612 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/04040d3e-3b48-47f2-8137-55ffd432390d-installation-pull-secrets\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.886119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.885683 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/04040d3e-3b48-47f2-8137-55ffd432390d-ca-trust-extracted\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.886119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.885768 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/04040d3e-3b48-47f2-8137-55ffd432390d-image-registry-private-configuration\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.886119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.885897 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04040d3e-3b48-47f2-8137-55ffd432390d-registry-tls\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.886119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.885930 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs7qx\" (UniqueName: \"kubernetes.io/projected/04040d3e-3b48-47f2-8137-55ffd432390d-kube-api-access-rs7qx\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.886119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.885969 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04040d3e-3b48-47f2-8137-55ffd432390d-trusted-ca\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.886119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.885999 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/04040d3e-3b48-47f2-8137-55ffd432390d-registry-certificates\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.942546 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.942408 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:18.970519 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.970489 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd" Sep 16 01:17:18.987255 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.987214 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04040d3e-3b48-47f2-8137-55ffd432390d-bound-sa-token\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.987367 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.987277 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/04040d3e-3b48-47f2-8137-55ffd432390d-installation-pull-secrets\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.987367 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.987308 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/04040d3e-3b48-47f2-8137-55ffd432390d-ca-trust-extracted\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.987367 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.987347 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/04040d3e-3b48-47f2-8137-55ffd432390d-image-registry-private-configuration\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.987535 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.987408 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04040d3e-3b48-47f2-8137-55ffd432390d-registry-tls\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.987535 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.987441 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs7qx\" (UniqueName: \"kubernetes.io/projected/04040d3e-3b48-47f2-8137-55ffd432390d-kube-api-access-rs7qx\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.987535 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.987483 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04040d3e-3b48-47f2-8137-55ffd432390d-trusted-ca\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.987535 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.987515 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/04040d3e-3b48-47f2-8137-55ffd432390d-registry-certificates\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.987813 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.987787 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/04040d3e-3b48-47f2-8137-55ffd432390d-ca-trust-extracted\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.988404 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.988361 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/04040d3e-3b48-47f2-8137-55ffd432390d-registry-certificates\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.988849 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.988816 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04040d3e-3b48-47f2-8137-55ffd432390d-trusted-ca\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.991135 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.991072 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/04040d3e-3b48-47f2-8137-55ffd432390d-image-registry-private-configuration\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.991446 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.991423 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/04040d3e-3b48-47f2-8137-55ffd432390d-installation-pull-secrets\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.993344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.993321 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04040d3e-3b48-47f2-8137-55ffd432390d-registry-tls\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:18.995753 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:18.995729 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04040d3e-3b48-47f2-8137-55ffd432390d-bound-sa-token\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:19.003611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.003572 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs7qx\" (UniqueName: \"kubernetes.io/projected/04040d3e-3b48-47f2-8137-55ffd432390d-kube-api-access-rs7qx\") pod \"image-registry-659c7b6595-x87bp\" (UID: \"04040d3e-3b48-47f2-8137-55ffd432390d\") " pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:19.078784 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.078744 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:19.083555 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.083523 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-79qw7\"" Sep 16 01:17:19.092500 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.092478 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:19.133092 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.133056 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:19.678660 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.678627 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:17:19.687841 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.687816 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:17:19.794603 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.794556 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4be6a433-aedd-4d7a-98da-58077fbed697-image-registry-private-configuration\") pod \"4be6a433-aedd-4d7a-98da-58077fbed697\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " Sep 16 01:17:19.794785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.794622 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4be6a433-aedd-4d7a-98da-58077fbed697-trusted-ca\") pod \"4be6a433-aedd-4d7a-98da-58077fbed697\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " Sep 16 01:17:19.794785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.794652 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4be6a433-aedd-4d7a-98da-58077fbed697-installation-pull-secrets\") pod \"4be6a433-aedd-4d7a-98da-58077fbed697\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " Sep 16 01:17:19.794785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.794688 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4be6a433-aedd-4d7a-98da-58077fbed697-ca-trust-extracted\") pod \"4be6a433-aedd-4d7a-98da-58077fbed697\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " Sep 16 01:17:19.794932 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.794827 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn9mq\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-kube-api-access-fn9mq\") pod \"4be6a433-aedd-4d7a-98da-58077fbed697\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " Sep 16 01:17:19.794932 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.794857 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4be6a433-aedd-4d7a-98da-58077fbed697-registry-certificates\") pod \"4be6a433-aedd-4d7a-98da-58077fbed697\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " Sep 16 01:17:19.794932 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.794898 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-bound-sa-token\") pod \"4be6a433-aedd-4d7a-98da-58077fbed697\" (UID: \"4be6a433-aedd-4d7a-98da-58077fbed697\") " Sep 16 01:17:19.795069 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.795022 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be6a433-aedd-4d7a-98da-58077fbed697-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4be6a433-aedd-4d7a-98da-58077fbed697" (UID: "4be6a433-aedd-4d7a-98da-58077fbed697"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:17:19.795712 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.795184 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be6a433-aedd-4d7a-98da-58077fbed697-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4be6a433-aedd-4d7a-98da-58077fbed697" (UID: "4be6a433-aedd-4d7a-98da-58077fbed697"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:17:19.795712 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.795208 2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4be6a433-aedd-4d7a-98da-58077fbed697-ca-trust-extracted\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:17:19.795712 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.795220 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be6a433-aedd-4d7a-98da-58077fbed697-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4be6a433-aedd-4d7a-98da-58077fbed697" (UID: "4be6a433-aedd-4d7a-98da-58077fbed697"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:17:19.797673 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.797629 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-kube-api-access-fn9mq" (OuterVolumeSpecName: "kube-api-access-fn9mq") pod "4be6a433-aedd-4d7a-98da-58077fbed697" (UID: "4be6a433-aedd-4d7a-98da-58077fbed697"). InnerVolumeSpecName "kube-api-access-fn9mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:17:19.797673 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.797634 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be6a433-aedd-4d7a-98da-58077fbed697-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4be6a433-aedd-4d7a-98da-58077fbed697" (UID: "4be6a433-aedd-4d7a-98da-58077fbed697"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:17:19.797817 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.797672 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4be6a433-aedd-4d7a-98da-58077fbed697" (UID: "4be6a433-aedd-4d7a-98da-58077fbed697"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:17:19.799083 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.799059 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be6a433-aedd-4d7a-98da-58077fbed697-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4be6a433-aedd-4d7a-98da-58077fbed697" (UID: "4be6a433-aedd-4d7a-98da-58077fbed697"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:17:19.896102 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.896062 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn9mq\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-kube-api-access-fn9mq\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:17:19.896102 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.896096 2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4be6a433-aedd-4d7a-98da-58077fbed697-registry-certificates\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:17:19.896102 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.896111 2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-bound-sa-token\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:17:19.896404 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.896124 2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4be6a433-aedd-4d7a-98da-58077fbed697-image-registry-private-configuration\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:17:19.896404 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.896136 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4be6a433-aedd-4d7a-98da-58077fbed697-trusted-ca\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:17:19.896404 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:19.896144 2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4be6a433-aedd-4d7a-98da-58077fbed697-installation-pull-secrets\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:17:20.682113 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:20.682076 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f9887d665-d8xxn" Sep 16 01:17:20.727598 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:20.727569 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f9887d665-d8xxn"] Sep 16 01:17:20.729948 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:20.729924 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f9887d665-d8xxn"] Sep 16 01:17:20.802222 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:20.802182 2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4be6a433-aedd-4d7a-98da-58077fbed697-registry-tls\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:17:21.448437 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:21.448361 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:22.292812 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.292773 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4be6a433-aedd-4d7a-98da-58077fbed697" path="/var/lib/kubelet/pods/4be6a433-aedd-4d7a-98da-58077fbed697/volumes" Sep 16 01:17:22.603649 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.603618 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd"] Sep 16 01:17:22.606819 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.606790 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-659c7b6595-x87bp"] Sep 16 01:17:22.608272 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:22.608245 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a4b0c50_3ab7_4d56_b612_0a2fcad22161.slice/crio-f3214e3e817f370195a4db073fdc0aa040d411ffcc530966f331147163a52164 WatchSource:0}: Error finding container f3214e3e817f370195a4db073fdc0aa040d411ffcc530966f331147163a52164: Status 404 returned error can't find the container with id f3214e3e817f370195a4db073fdc0aa040d411ffcc530966f331147163a52164 Sep 16 01:17:22.618321 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:22.618293 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04040d3e_3b48_47f2_8137_55ffd432390d.slice/crio-82f2905bb43de9ac85c4247125694beebb4699f02b7f4db1ec94ee8d1a848ffe WatchSource:0}: Error finding container 82f2905bb43de9ac85c4247125694beebb4699f02b7f4db1ec94ee8d1a848ffe: Status 404 returned error can't find the container with id 82f2905bb43de9ac85c4247125694beebb4699f02b7f4db1ec94ee8d1a848ffe Sep 16 01:17:22.696113 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.696083 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd" event={"ID":"2a4b0c50-3ab7-4d56-b612-0a2fcad22161","Type":"ContainerStarted","Data":"f3214e3e817f370195a4db073fdc0aa040d411ffcc530966f331147163a52164"} Sep 16 01:17:22.697712 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.697685 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6ndbn" event={"ID":"5826c82e-35c3-40b5-a3f8-774437001cf8","Type":"ContainerStarted","Data":"0b040708e6f682d7ef3fc082a6f21d22eef31d49df45c8a7c171ba6fdb58d683"} Sep 16 01:17:22.699271 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.699248 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q6xx7" event={"ID":"afe22dac-90f9-4893-9e84-0d7731b9fd8f","Type":"ContainerStarted","Data":"b7e1f91a9301487a373877be137f5def746fa6234e47946dd0706fc21801f5d0"} Sep 16 01:17:22.699391 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.699313 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:17:22.701009 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.700988 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659c7b6595-x87bp" event={"ID":"04040d3e-3b48-47f2-8137-55ffd432390d","Type":"ContainerStarted","Data":"82f2905bb43de9ac85c4247125694beebb4699f02b7f4db1ec94ee8d1a848ffe"} Sep 16 01:17:22.703080 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.703062 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnb9f" event={"ID":"36024d3d-ea39-4326-b503-4d8a07691efa","Type":"ContainerStarted","Data":"718fd7d7b9590c2be999a046c6ebf920ee428380606030b9763bb94016272282"} Sep 16 01:17:22.703189 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.703095 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnb9f" event={"ID":"36024d3d-ea39-4326-b503-4d8a07691efa","Type":"ContainerStarted","Data":"fb121777e3ec7a707da3ab6f68b71b9e3b032bd67e2d62161f279ade61977f16"} Sep 16 01:17:22.703246 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.703206 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lnb9f" Sep 16 01:17:22.710713 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.710658 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6ndbn" podStartSLOduration=66.939863547 podStartE2EDuration="1m12.710641224s" totalImagesPullingTime="5.770777677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:22.709955662 +0000 UTC m=+72.962378250" watchObservedRunningTime="2026-09-16 01:17:22.710641224 +0000 UTC m=+72.963063813" Sep 16 01:17:22.725460 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.725417 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q6xx7" podStartSLOduration=66.883974076 podStartE2EDuration="1m12.725403124s" totalImagesPullingTime="5.841429048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:22.724705597 +0000 UTC m=+72.977128188" watchObservedRunningTime="2026-09-16 01:17:22.725403124 +0000 UTC m=+72.977825712" Sep 16 01:17:22.739017 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:22.738975 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lnb9f" podStartSLOduration=32.884067351 podStartE2EDuration="37.738957306s" totalImagesPullingTime="4.854889955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:22.738817872 +0000 UTC m=+72.991240463" watchObservedRunningTime="2026-09-16 01:17:22.738957306 +0000 UTC m=+72.991379898" Sep 16 01:17:23.707820 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:23.707781 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659c7b6595-x87bp" event={"ID":"04040d3e-3b48-47f2-8137-55ffd432390d","Type":"ContainerStarted","Data":"92250c511cfb144119a2a97e6c88f55c87f8d7372928c4a386889b4f2c4f3021"} Sep 16 01:17:23.708247 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:23.707968 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:23.727304 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:23.727247 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-659c7b6595-x87bp" podStartSLOduration=5.727228871 podStartE2EDuration="5.727228871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:23.7264814 +0000 UTC m=+73.978903988" watchObservedRunningTime="2026-09-16 01:17:23.727228871 +0000 UTC m=+73.979651461" Sep 16 01:17:23.749599 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:23.749562 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:23.803148 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:23.803123 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:25.471373 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:25.471338 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:25.575681 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:25.575654 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:25.638514 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:25.638482 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd"] Sep 16 01:17:25.652230 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:25.652201 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:25.661936 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:25.661909 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd"] Sep 16 01:17:25.662015 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:25.662002 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd" Sep 16 01:17:25.664258 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:25.664228 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 16 01:17:25.664362 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:25.664290 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sspj4\"" Sep 16 01:17:25.734569 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:25.734515 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:25.736647 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:25.736625 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2ca52899-0cc2-4d38-942f-3659d1113017-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-x9kmd\" (UID: \"2ca52899-0cc2-4d38-942f-3659d1113017\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd" Sep 16 01:17:25.837044 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:25.837015 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2ca52899-0cc2-4d38-942f-3659d1113017-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-x9kmd\" (UID: \"2ca52899-0cc2-4d38-942f-3659d1113017\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd" Sep 16 01:17:25.837185 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:25.837152 2558 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 16 01:17:25.837240 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:25.837230 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca52899-0cc2-4d38-942f-3659d1113017-tls-certificates podName:2ca52899-0cc2-4d38-942f-3659d1113017 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:26.337212488 +0000 UTC m=+76.589635077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/2ca52899-0cc2-4d38-942f-3659d1113017-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-x9kmd" (UID: "2ca52899-0cc2-4d38-942f-3659d1113017") : secret "prometheus-operator-admission-webhook-tls" not found Sep 16 01:17:26.139143 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:26.139118 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:17:26.141382 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:26.141366 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9633a4e7-63e6-44e1-9469-7334c5dc56e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rrxxj\" (UID: \"9633a4e7-63e6-44e1-9469-7334c5dc56e4\") " pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:17:26.288376 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:26.288343 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:26.340389 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:26.340367 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2ca52899-0cc2-4d38-942f-3659d1113017-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-x9kmd\" (UID: \"2ca52899-0cc2-4d38-942f-3659d1113017\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd" Sep 16 01:17:26.342679 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:26.342655 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2ca52899-0cc2-4d38-942f-3659d1113017-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-x9kmd\" (UID: \"2ca52899-0cc2-4d38-942f-3659d1113017\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd" Sep 16 01:17:26.356388 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:26.356371 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ncdjs\"" Sep 16 01:17:26.364983 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:26.364969 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rrxxj" Sep 16 01:17:26.571567 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:26.571528 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd" Sep 16 01:17:26.634114 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:26.634088 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rrxxj"] Sep 16 01:17:26.637572 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:26.637527 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9633a4e7_63e6_44e1_9469_7334c5dc56e4.slice/crio-8b844216f7d6ba6b952f4641fb186016abc10ee7e31c9ab6fbde595a7c16c93f WatchSource:0}: Error finding container 8b844216f7d6ba6b952f4641fb186016abc10ee7e31c9ab6fbde595a7c16c93f: Status 404 returned error can't find the container with id 8b844216f7d6ba6b952f4641fb186016abc10ee7e31c9ab6fbde595a7c16c93f Sep 16 01:17:26.722950 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:26.722920 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rrxxj" event={"ID":"9633a4e7-63e6-44e1-9469-7334c5dc56e4","Type":"ContainerStarted","Data":"4d098815556e7d2d8c097d402e68fe866827a1ced7d449730291761ea4cf1bb9"} Sep 16 01:17:26.723064 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:26.722980 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rrxxj" event={"ID":"9633a4e7-63e6-44e1-9469-7334c5dc56e4","Type":"ContainerStarted","Data":"8b844216f7d6ba6b952f4641fb186016abc10ee7e31c9ab6fbde595a7c16c93f"} Sep 16 01:17:26.838499 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:26.838469 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd"] Sep 16 01:17:26.841217 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:26.841192 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca52899_0cc2_4d38_942f_3659d1113017.slice/crio-0a7cc88fb1b3f22732d1b7ebdd12dde25e96306fc91910d41cecbae6188b044b WatchSource:0}: Error finding container 0a7cc88fb1b3f22732d1b7ebdd12dde25e96306fc91910d41cecbae6188b044b: Status 404 returned error can't find the container with id 0a7cc88fb1b3f22732d1b7ebdd12dde25e96306fc91910d41cecbae6188b044b Sep 16 01:17:27.728798 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:27.728769 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rrxxj" event={"ID":"9633a4e7-63e6-44e1-9469-7334c5dc56e4","Type":"ContainerStarted","Data":"397ebc5e60e51ef524b4b8eb1db32c7898867df9e4f9945c95cfbb03399136d5"} Sep 16 01:17:27.730252 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:27.730227 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd" event={"ID":"2ca52899-0cc2-4d38-942f-3659d1113017","Type":"ContainerStarted","Data":"0a7cc88fb1b3f22732d1b7ebdd12dde25e96306fc91910d41cecbae6188b044b"} Sep 16 01:17:28.735343 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:28.735305 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd" event={"ID":"2ca52899-0cc2-4d38-942f-3659d1113017","Type":"ContainerStarted","Data":"89a43ce5e3a11eec81cbbd84c6a9b1fccd1a0c6ae96a8ad5065793cefd13e01d"} Sep 16 01:17:28.735749 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:28.735554 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd" Sep 16 01:17:28.740718 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:28.740694 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd" Sep 16 01:17:28.750265 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:28.750225 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-x9kmd" podStartSLOduration=2.606283511 podStartE2EDuration="3.750209125s" totalImagesPullingTime="1.143925614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:28.749051557 +0000 UTC m=+79.001474147" watchObservedRunningTime="2026-09-16 01:17:28.750209125 +0000 UTC m=+79.002631714" Sep 16 01:17:29.660508 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:29.660478 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:29.729150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:29.729115 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:29.740441 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:29.740410 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rrxxj" event={"ID":"9633a4e7-63e6-44e1-9469-7334c5dc56e4","Type":"ContainerStarted","Data":"be1f799c1ceda0a50602eccbc26506e4d171a46d4cf586f26111896fd9f4c07c"} Sep 16 01:17:29.756848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:29.756805 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rrxxj" podStartSLOduration=33.417511534 podStartE2EDuration="35.756791993s" totalImagesPullingTime="2.339280459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:16:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:29.756510535 +0000 UTC m=+80.008933124" watchObservedRunningTime="2026-09-16 01:17:29.756791993 +0000 UTC m=+80.009214582" Sep 16 01:17:29.932619 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:29.932552 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:31.458146 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.458024 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77d9cf858b-6jstg"] Sep 16 01:17:31.461917 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.461888 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.464381 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.464361 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 01:17:31.464497 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.464363 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 01:17:31.464979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.464960 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 01:17:31.465093 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.464961 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 01:17:31.465093 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.465084 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 01:17:31.465254 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.465105 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 01:17:31.465407 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.465393 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 01:17:31.465469 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.465450 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dwcgh\"" Sep 16 01:17:31.469539 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.469522 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 01:17:31.472563 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.472541 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d9cf858b-6jstg"] Sep 16 01:17:31.477737 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.477713 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:31.536856 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.536832 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:31.576084 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.576060 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-oauth-serving-cert\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.576194 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.576090 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aee7b430-f393-464c-bb00-4e978de04450-console-serving-cert\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.576194 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.576111 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-console-config\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.576194 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.576180 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-trusted-ca-bundle\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.576296 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.576217 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aee7b430-f393-464c-bb00-4e978de04450-console-oauth-config\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.576296 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.576239 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-service-ca\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.576296 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.576257 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6bvb\" (UniqueName: \"kubernetes.io/projected/aee7b430-f393-464c-bb00-4e978de04450-kube-api-access-j6bvb\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.676540 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.676516 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-console-config\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.676638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.676550 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-trusted-ca-bundle\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.676638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.676569 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aee7b430-f393-464c-bb00-4e978de04450-console-oauth-config\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.676638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.676595 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-service-ca\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.676638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.676611 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6bvb\" (UniqueName: \"kubernetes.io/projected/aee7b430-f393-464c-bb00-4e978de04450-kube-api-access-j6bvb\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.676831 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.676640 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-oauth-serving-cert\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.676831 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.676664 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aee7b430-f393-464c-bb00-4e978de04450-console-serving-cert\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.677356 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.677335 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-service-ca\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.677464 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.677429 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-console-config\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.677662 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.677640 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-oauth-serving-cert\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.677848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.677824 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-trusted-ca-bundle\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.678959 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.678936 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aee7b430-f393-464c-bb00-4e978de04450-console-oauth-config\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.679098 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.679082 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aee7b430-f393-464c-bb00-4e978de04450-console-serving-cert\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.684583 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.684563 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6bvb\" (UniqueName: \"kubernetes.io/projected/aee7b430-f393-464c-bb00-4e978de04450-kube-api-access-j6bvb\") pod \"console-77d9cf858b-6jstg\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:31.771489 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:31.771433 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:32.044548 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:32.044464 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d9cf858b-6jstg"] Sep 16 01:17:32.046693 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:32.046665 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee7b430_f393_464c_bb00_4e978de04450.slice/crio-205f86fc22a9f1cd6ea5c95cee3b88d5de6b586708c4f5ef78afa10b5691b7a1 WatchSource:0}: Error finding container 205f86fc22a9f1cd6ea5c95cee3b88d5de6b586708c4f5ef78afa10b5691b7a1: Status 404 returned error can't find the container with id 205f86fc22a9f1cd6ea5c95cee3b88d5de6b586708c4f5ef78afa10b5691b7a1 Sep 16 01:17:32.452568 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:32.452537 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:32.552942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:32.552912 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:32.710739 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:32.710669 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lnb9f" Sep 16 01:17:32.752116 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:32.752079 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d9cf858b-6jstg" event={"ID":"aee7b430-f393-464c-bb00-4e978de04450","Type":"ContainerStarted","Data":"205f86fc22a9f1cd6ea5c95cee3b88d5de6b586708c4f5ef78afa10b5691b7a1"} Sep 16 01:17:35.458033 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:35.457998 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:35.555204 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:35.555170 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:35.765916 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:35.765833 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d9cf858b-6jstg" event={"ID":"aee7b430-f393-464c-bb00-4e978de04450","Type":"ContainerStarted","Data":"6ee6393b8c446a67b33cbd7b0a05358621f1b5ed1dc08bdf3722260ee39b4ac2"} Sep 16 01:17:35.782143 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:35.782085 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77d9cf858b-6jstg" podStartSLOduration=2.012502551 podStartE2EDuration="4.782064989s" totalImagesPullingTime="2.769562438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:35.781137406 +0000 UTC m=+86.033560007" watchObservedRunningTime="2026-09-16 01:17:35.782064989 +0000 UTC m=+86.034487578" Sep 16 01:17:37.774892 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:37.774854 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd" event={"ID":"2a4b0c50-3ab7-4d56-b612-0a2fcad22161","Type":"ContainerStarted","Data":"19ee9e14c9a2c3e12490150b810be94869ddbc1d6b66908b491304f1569df1a8"} Sep 16 01:17:37.789340 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:37.789296 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5bdc8f79fd-4rzfd" podStartSLOduration=5.506558346 podStartE2EDuration="19.789282317s" totalImagesPullingTime="14.282723971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:37.788454775 +0000 UTC m=+88.040877375" watchObservedRunningTime="2026-09-16 01:17:37.789282317 +0000 UTC m=+88.041704907" Sep 16 01:17:39.926435 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:39.926404 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:41.060277 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:41.060228 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:41.109607 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:41.109575 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:41.772056 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:41.772023 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:41.772276 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:41.772096 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:41.776798 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:41.776776 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:41.790452 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:41.790428 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:17:41.809118 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:41.809084 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:41.909387 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:41.909338 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:42.010699 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.010667 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:42.060356 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.060294 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4"] Sep 16 01:17:42.063269 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.063247 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.063834 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.063811 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vh6bv"] Sep 16 01:17:42.065390 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.065367 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 01:17:42.065578 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.065561 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 16 01:17:42.065847 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.065824 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8j7dt\"" Sep 16 01:17:42.065847 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.065837 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 16 01:17:42.066012 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.065856 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 01:17:42.066012 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.065846 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 01:17:42.066369 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.066351 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 16 01:17:42.066785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.066771 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.069437 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.069417 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 01:17:42.069589 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.069567 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bnfrk\"" Sep 16 01:17:42.069695 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.069461 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 01:17:42.069695 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.069441 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 01:17:42.077267 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.077242 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4"] Sep 16 01:17:42.099440 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.099407 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:42.151003 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.150969 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.151110 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151009 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e7b745db-d94b-428e-8012-cab5a3d44d1d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.151110 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151062 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.151110 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151090 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-sys\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.151256 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151117 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-root\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.151256 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151142 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.151256 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151171 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfdx6\" (UniqueName: \"kubernetes.io/projected/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-api-access-mfdx6\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.151256 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151209 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-textfile\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.151256 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151241 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzczc\" (UniqueName: \"kubernetes.io/projected/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-kube-api-access-vzczc\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.151256 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151256 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7b745db-d94b-428e-8012-cab5a3d44d1d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.151497 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151296 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-tls\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.151497 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151343 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-wtmp\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.151497 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151362 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.151497 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151382 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-metrics-client-ca\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.151497 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.151400 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.200132 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.200104 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:42.252451 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252427 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e7b745db-d94b-428e-8012-cab5a3d44d1d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.252552 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252466 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.252552 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252484 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-sys\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.252552 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252503 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-root\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.252552 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252527 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.252552 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252549 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfdx6\" (UniqueName: \"kubernetes.io/projected/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-api-access-mfdx6\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.252804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252565 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-sys\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.252804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252565 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-root\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.252804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252569 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-textfile\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.252804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252643 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzczc\" (UniqueName: \"kubernetes.io/projected/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-kube-api-access-vzczc\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.252804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252667 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7b745db-d94b-428e-8012-cab5a3d44d1d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.252804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252689 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-tls\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.252804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252718 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-wtmp\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.252804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252744 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.252804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252777 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-metrics-client-ca\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.252804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252805 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.253335 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252839 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.253335 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252872 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e7b745db-d94b-428e-8012-cab5a3d44d1d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.253335 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252925 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-textfile\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.253335 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.252941 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-wtmp\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.253335 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:42.253028 2558 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 16 01:17:42.253335 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:42.253092 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-tls podName:e7b745db-d94b-428e-8012-cab5a3d44d1d nodeName:}" failed. No retries permitted until 2026-09-16 01:17:42.753074409 +0000 UTC m=+93.005496976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-gcbb4" (UID: "e7b745db-d94b-428e-8012-cab5a3d44d1d") : secret "kube-state-metrics-tls" not found Sep 16 01:17:42.253335 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.253299 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.253697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.253480 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7b745db-d94b-428e-8012-cab5a3d44d1d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.253697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.253487 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-metrics-client-ca\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.253697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.253487 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.255389 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.255368 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.255516 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.255498 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.255573 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.255527 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-node-exporter-tls\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.260038 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.260014 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfdx6\" (UniqueName: \"kubernetes.io/projected/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-api-access-mfdx6\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.260319 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.260302 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzczc\" (UniqueName: \"kubernetes.io/projected/af3a00ef-4f24-41c6-8ec5-65f9fc8685ac-kube-api-access-vzczc\") pod \"node-exporter-vh6bv\" (UID: \"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac\") " pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.325548 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.325480 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:42.379795 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.379768 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vh6bv" Sep 16 01:17:42.391288 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:42.391265 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3a00ef_4f24_41c6_8ec5_65f9fc8685ac.slice/crio-9d842ef685a0e5905b68c1e3614199ed3ba4e5d8e4b52a77dfadc1ce75509b83 WatchSource:0}: Error finding container 9d842ef685a0e5905b68c1e3614199ed3ba4e5d8e4b52a77dfadc1ce75509b83: Status 404 returned error can't find the container with id 9d842ef685a0e5905b68c1e3614199ed3ba4e5d8e4b52a77dfadc1ce75509b83 Sep 16 01:17:42.616495 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.616468 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:42.717465 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.717432 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:42.757755 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.757728 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:42.757882 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:42.757864 2558 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 16 01:17:42.757929 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:17:42.757920 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-tls podName:e7b745db-d94b-428e-8012-cab5a3d44d1d nodeName:}" failed. No retries permitted until 2026-09-16 01:17:43.757907587 +0000 UTC m=+94.010330158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-gcbb4" (UID: "e7b745db-d94b-428e-8012-cab5a3d44d1d") : secret "kube-state-metrics-tls" not found Sep 16 01:17:42.791662 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.791639 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh6bv" event={"ID":"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac","Type":"ContainerStarted","Data":"9d842ef685a0e5905b68c1e3614199ed3ba4e5d8e4b52a77dfadc1ce75509b83"} Sep 16 01:17:42.895384 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.895310 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:42.987273 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:42.987241 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:43.087791 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:43.087746 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:43.190037 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:43.189954 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:43.489731 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:43.489665 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:43.590058 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:43.590034 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:43.767060 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:43.767000 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:43.769340 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:43.769316 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7b745db-d94b-428e-8012-cab5a3d44d1d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-gcbb4\" (UID: \"e7b745db-d94b-428e-8012-cab5a3d44d1d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:43.796558 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:43.796529 2558 generic.go:356] "Generic (PLEG): container finished" podID="af3a00ef-4f24-41c6-8ec5-65f9fc8685ac" containerID="a1197857f65b99f734fc652e4bfc2f0fc64b652bfb9857711684cc779f8b9ef1" exitCode=0 Sep 16 01:17:43.796664 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:43.796576 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh6bv" event={"ID":"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac","Type":"ContainerDied","Data":"a1197857f65b99f734fc652e4bfc2f0fc64b652bfb9857711684cc779f8b9ef1"} Sep 16 01:17:43.873404 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:43.873381 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" Sep 16 01:17:44.145055 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:44.144888 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4"] Sep 16 01:17:44.147575 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:44.147549 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b745db_d94b_428e_8012_cab5a3d44d1d.slice/crio-527559dac4663a2c9a78a9abf85adf0fbb62c30607acceb8097815526d9dc0ae WatchSource:0}: Error finding container 527559dac4663a2c9a78a9abf85adf0fbb62c30607acceb8097815526d9dc0ae: Status 404 returned error can't find the container with id 527559dac4663a2c9a78a9abf85adf0fbb62c30607acceb8097815526d9dc0ae Sep 16 01:17:44.715559 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:44.715528 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-659c7b6595-x87bp" Sep 16 01:17:44.804314 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:44.804278 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh6bv" event={"ID":"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac","Type":"ContainerStarted","Data":"37f5e2af690a697489d485af506b8f8c968b91087d333cc03992e244d38865bb"} Sep 16 01:17:44.804496 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:44.804344 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh6bv" event={"ID":"af3a00ef-4f24-41c6-8ec5-65f9fc8685ac","Type":"ContainerStarted","Data":"250a3638e0691e0c9a0c601549b45f6dbd0f9614655d0ef2e08d2a9725dca60f"} Sep 16 01:17:44.806045 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:44.806008 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" event={"ID":"e7b745db-d94b-428e-8012-cab5a3d44d1d","Type":"ContainerStarted","Data":"527559dac4663a2c9a78a9abf85adf0fbb62c30607acceb8097815526d9dc0ae"} Sep 16 01:17:44.822764 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:44.822720 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vh6bv" podStartSLOduration=1.773292413 podStartE2EDuration="2.822704231s" totalImagesPullingTime="980.728551ms" totalInitContainerRuntime="68.683267ms" isStatefulPod=true podCreationTimestamp="2026-09-16 01:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:44.820673784 +0000 UTC m=+95.073096374" watchObservedRunningTime="2026-09-16 01:17:44.822704231 +0000 UTC m=+95.075126820" Sep 16 01:17:44.843543 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:44.843510 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:44.941321 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:44.941264 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:45.044649 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.044542 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:45.044805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.044645 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp"] Sep 16 01:17:45.048342 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.048323 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.050902 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.050883 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7pptb\"" Sep 16 01:17:45.051002 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.050905 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 16 01:17:45.051002 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.050907 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 16 01:17:45.051002 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.050903 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 16 01:17:45.051002 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.050956 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 16 01:17:45.051243 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.051228 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 16 01:17:45.051366 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.051343 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3ni8vqtsarrct\"" Sep 16 01:17:45.059409 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.059384 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp"] Sep 16 01:17:45.161751 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.161718 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:45.178904 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.178871 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqk2\" (UniqueName: \"kubernetes.io/projected/a734a692-d38a-4b2b-b932-de84fea3dceb-kube-api-access-ztqk2\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.179058 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.178911 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.179058 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.178935 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-tls\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.179058 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.178963 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.179226 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.179068 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a734a692-d38a-4b2b-b932-de84fea3dceb-metrics-client-ca\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.179226 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.179107 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-grpc-tls\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.179226 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.179139 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.179341 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.179230 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.208615 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.208571 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:45.262114 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.262069 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:45.279652 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.279618 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqk2\" (UniqueName: \"kubernetes.io/projected/a734a692-d38a-4b2b-b932-de84fea3dceb-kube-api-access-ztqk2\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.279785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.279661 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.279785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.279686 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-tls\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.279785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.279712 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.279785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.279755 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a734a692-d38a-4b2b-b932-de84fea3dceb-metrics-client-ca\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.279785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.279784 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-grpc-tls\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.280110 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.279810 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.280110 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.279878 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.280678 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.280652 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a734a692-d38a-4b2b-b932-de84fea3dceb-metrics-client-ca\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.283204 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.283183 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.283289 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.283193 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-tls\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.283289 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.283255 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.283408 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.283322 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.283459 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.283434 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-grpc-tls\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.284126 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.284101 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a734a692-d38a-4b2b-b932-de84fea3dceb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.290425 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.290406 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqk2\" (UniqueName: \"kubernetes.io/projected/a734a692-d38a-4b2b-b932-de84fea3dceb-kube-api-access-ztqk2\") pod \"thanos-querier-7b4c7d77c7-wjvsp\" (UID: \"a734a692-d38a-4b2b-b932-de84fea3dceb\") " pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.358656 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.358585 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:45.696380 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.696355 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp"] Sep 16 01:17:45.700358 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:45.700330 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda734a692_d38a_4b2b_b932_de84fea3dceb.slice/crio-ea92ae23152c6a591114666cf0fac91ebf42feddcacd58c1f532a3ca772ad3b4 WatchSource:0}: Error finding container ea92ae23152c6a591114666cf0fac91ebf42feddcacd58c1f532a3ca772ad3b4: Status 404 returned error can't find the container with id ea92ae23152c6a591114666cf0fac91ebf42feddcacd58c1f532a3ca772ad3b4 Sep 16 01:17:45.814308 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.814276 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" event={"ID":"e7b745db-d94b-428e-8012-cab5a3d44d1d","Type":"ContainerStarted","Data":"09c3da38c693f0344ffbbf94d3cb39627549b4183d661301a688bbdfb106939c"} Sep 16 01:17:45.814482 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.814332 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" event={"ID":"e7b745db-d94b-428e-8012-cab5a3d44d1d","Type":"ContainerStarted","Data":"a9c06a1ebe63feada6f57f66b682f45c1b124aeb352b4bd7aacd3c3c05f73219"} Sep 16 01:17:45.814482 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.814369 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" event={"ID":"e7b745db-d94b-428e-8012-cab5a3d44d1d","Type":"ContainerStarted","Data":"a5c9489479f80943693beb9635b8a6aebc843aa11243d89c4ecf250d204857a3"} Sep 16 01:17:45.815701 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.815670 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" event={"ID":"a734a692-d38a-4b2b-b932-de84fea3dceb","Type":"ContainerStarted","Data":"ea92ae23152c6a591114666cf0fac91ebf42feddcacd58c1f532a3ca772ad3b4"} Sep 16 01:17:45.831467 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:45.831424 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-gcbb4" podStartSLOduration=2.508340972 podStartE2EDuration="3.831405376s" totalImagesPullingTime="1.323064404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:45.829755651 +0000 UTC m=+96.082178242" watchObservedRunningTime="2026-09-16 01:17:45.831405376 +0000 UTC m=+96.083827965" Sep 16 01:17:46.345299 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.345262 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6496c56f47-t8xsm"] Sep 16 01:17:46.348479 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.348464 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.350899 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.350871 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 16 01:17:46.351028 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.350909 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 16 01:17:46.351028 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.350983 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 16 01:17:46.351703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.351684 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-s2b175cer3t4\"" Sep 16 01:17:46.351785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.351722 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 16 01:17:46.351785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.351740 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-b4km7\"" Sep 16 01:17:46.359585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.359555 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6496c56f47-t8xsm"] Sep 16 01:17:46.490840 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.490810 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad694c3c-8319-4a83-a520-a6a13c1a077e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.490988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.490856 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ad694c3c-8319-4a83-a520-a6a13c1a077e-secret-metrics-server-tls\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.490988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.490889 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ad694c3c-8319-4a83-a520-a6a13c1a077e-audit-log\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.491102 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.490985 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ad694c3c-8319-4a83-a520-a6a13c1a077e-secret-metrics-server-client-certs\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.491102 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.491021 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ad694c3c-8319-4a83-a520-a6a13c1a077e-client-ca-bundle\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.491243 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.491106 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ad694c3c-8319-4a83-a520-a6a13c1a077e-metrics-server-audit-profiles\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.491243 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.491131 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw4mm\" (UniqueName: \"kubernetes.io/projected/ad694c3c-8319-4a83-a520-a6a13c1a077e-kube-api-access-cw4mm\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.591950 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.591918 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad694c3c-8319-4a83-a520-a6a13c1a077e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.591950 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.591952 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ad694c3c-8319-4a83-a520-a6a13c1a077e-secret-metrics-server-tls\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.592126 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.591969 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ad694c3c-8319-4a83-a520-a6a13c1a077e-audit-log\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.592126 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.592090 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ad694c3c-8319-4a83-a520-a6a13c1a077e-secret-metrics-server-client-certs\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.592126 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.592119 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ad694c3c-8319-4a83-a520-a6a13c1a077e-client-ca-bundle\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.592264 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.592208 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ad694c3c-8319-4a83-a520-a6a13c1a077e-metrics-server-audit-profiles\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.592264 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.592234 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw4mm\" (UniqueName: \"kubernetes.io/projected/ad694c3c-8319-4a83-a520-a6a13c1a077e-kube-api-access-cw4mm\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.592399 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.592381 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ad694c3c-8319-4a83-a520-a6a13c1a077e-audit-log\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.592768 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.592742 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad694c3c-8319-4a83-a520-a6a13c1a077e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.593134 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.593112 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ad694c3c-8319-4a83-a520-a6a13c1a077e-metrics-server-audit-profiles\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.594467 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.594448 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ad694c3c-8319-4a83-a520-a6a13c1a077e-secret-metrics-server-tls\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.595056 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.595028 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ad694c3c-8319-4a83-a520-a6a13c1a077e-client-ca-bundle\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.595228 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.595206 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ad694c3c-8319-4a83-a520-a6a13c1a077e-secret-metrics-server-client-certs\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.599386 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.599348 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw4mm\" (UniqueName: \"kubernetes.io/projected/ad694c3c-8319-4a83-a520-a6a13c1a077e-kube-api-access-cw4mm\") pod \"metrics-server-6496c56f47-t8xsm\" (UID: \"ad694c3c-8319-4a83-a520-a6a13c1a077e\") " pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.663591 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.663554 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:17:46.833386 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.832316 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-887c86869-ftklv"] Sep 16 01:17:46.837039 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.837009 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:46.837830 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.837807 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:46.865470 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.865440 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-887c86869-ftklv"] Sep 16 01:17:46.927721 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.927692 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:46.944377 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.944345 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6496c56f47-t8xsm"] Sep 16 01:17:46.948529 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:46.948502 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad694c3c_8319_4a83_a520_a6a13c1a077e.slice/crio-fe21b28d5f4355ab4cefc0b60b5cd3d6a5c83bffd71d2d1e475fd1f413343809 WatchSource:0}: Error finding container fe21b28d5f4355ab4cefc0b60b5cd3d6a5c83bffd71d2d1e475fd1f413343809: Status 404 returned error can't find the container with id fe21b28d5f4355ab4cefc0b60b5cd3d6a5c83bffd71d2d1e475fd1f413343809 Sep 16 01:17:46.996613 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.996580 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-trusted-ca-bundle\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:46.996764 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.996676 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/144e3b20-3f4b-45f6-ac63-31371e640363-console-oauth-config\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:46.996764 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.996727 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cpml\" (UniqueName: \"kubernetes.io/projected/144e3b20-3f4b-45f6-ac63-31371e640363-kube-api-access-7cpml\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:46.996852 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.996766 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-service-ca\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:46.996852 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.996787 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-oauth-serving-cert\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:46.996852 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.996812 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-console-config\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:46.997005 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:46.996880 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/144e3b20-3f4b-45f6-ac63-31371e640363-console-serving-cert\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.042688 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.042651 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:47.097867 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.097840 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cpml\" (UniqueName: \"kubernetes.io/projected/144e3b20-3f4b-45f6-ac63-31371e640363-kube-api-access-7cpml\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.097981 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.097880 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-service-ca\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.097981 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.097899 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-oauth-serving-cert\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.097981 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.097929 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-console-config\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.097981 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.097962 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/144e3b20-3f4b-45f6-ac63-31371e640363-console-serving-cert\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.098204 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.098006 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-trusted-ca-bundle\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.098204 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.098039 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/144e3b20-3f4b-45f6-ac63-31371e640363-console-oauth-config\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.098690 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.098653 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-service-ca\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.098797 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.098732 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-oauth-serving-cert\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.098797 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.098767 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-console-config\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.099267 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.099243 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-trusted-ca-bundle\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.100463 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.100438 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/144e3b20-3f4b-45f6-ac63-31371e640363-console-serving-cert\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.100590 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.100573 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/144e3b20-3f4b-45f6-ac63-31371e640363-console-oauth-config\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.104539 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.104522 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cpml\" (UniqueName: \"kubernetes.io/projected/144e3b20-3f4b-45f6-ac63-31371e640363-kube-api-access-7cpml\") pod \"console-887c86869-ftklv\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.148800 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.148694 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:47.187146 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.187114 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:47.423479 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.423405 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-887c86869-ftklv"] Sep 16 01:17:47.827984 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:47.827897 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" event={"ID":"ad694c3c-8319-4a83-a520-a6a13c1a077e","Type":"ContainerStarted","Data":"fe21b28d5f4355ab4cefc0b60b5cd3d6a5c83bffd71d2d1e475fd1f413343809"} Sep 16 01:17:47.887911 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:17:47.887877 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144e3b20_3f4b_45f6_ac63_31371e640363.slice/crio-83e8f8d818603ae43514d5c750549c6ca76715914212cf6bbe3c988cc32f9d96 WatchSource:0}: Error finding container 83e8f8d818603ae43514d5c750549c6ca76715914212cf6bbe3c988cc32f9d96: Status 404 returned error can't find the container with id 83e8f8d818603ae43514d5c750549c6ca76715914212cf6bbe3c988cc32f9d96 Sep 16 01:17:48.263489 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:48.263457 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:48.834202 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:48.834149 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-887c86869-ftklv" event={"ID":"144e3b20-3f4b-45f6-ac63-31371e640363","Type":"ContainerStarted","Data":"4a1d41e8dd2ffa93a3fcd0e4eb1c2cd2cd1e75771700f009d7a0115442c90ca2"} Sep 16 01:17:48.834654 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:48.834217 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-887c86869-ftklv" event={"ID":"144e3b20-3f4b-45f6-ac63-31371e640363","Type":"ContainerStarted","Data":"83e8f8d818603ae43514d5c750549c6ca76715914212cf6bbe3c988cc32f9d96"} Sep 16 01:17:48.836688 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:48.836659 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" event={"ID":"a734a692-d38a-4b2b-b932-de84fea3dceb","Type":"ContainerStarted","Data":"b0a7c293a2c09e44fe4473f50a44c7059f382c5e5b645cbb29fbc58737622073"} Sep 16 01:17:48.836799 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:48.836718 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" event={"ID":"a734a692-d38a-4b2b-b932-de84fea3dceb","Type":"ContainerStarted","Data":"41fdc29ea741643b7d881d7a3fff9fd2e04e63e0c7031ec8b5d3735b375b5a66"} Sep 16 01:17:48.836799 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:48.836744 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" event={"ID":"a734a692-d38a-4b2b-b932-de84fea3dceb","Type":"ContainerStarted","Data":"d94fb338f20a0f37c0a0058084cf469124c42559961971107c7754bb02886c94"} Sep 16 01:17:48.851215 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:48.851150 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-887c86869-ftklv" podStartSLOduration=2.851128929 podStartE2EDuration="2.851128929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:48.849970263 +0000 UTC m=+99.102392853" watchObservedRunningTime="2026-09-16 01:17:48.851128929 +0000 UTC m=+99.103551512" Sep 16 01:17:49.844136 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:49.844058 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" event={"ID":"a734a692-d38a-4b2b-b932-de84fea3dceb","Type":"ContainerStarted","Data":"eeeda0066f16e39a40b1a99e7fa2a838083e535e3ab73c2bb8d8a31c03fa6174"} Sep 16 01:17:49.844635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:49.844216 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:49.844635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:49.844267 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" event={"ID":"a734a692-d38a-4b2b-b932-de84fea3dceb","Type":"ContainerStarted","Data":"768f6c2108261161b873f6e0b8e6febdf6e2b80b52ed0eabc8e63898fcd61d2d"} Sep 16 01:17:49.844635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:49.844317 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" event={"ID":"a734a692-d38a-4b2b-b932-de84fea3dceb","Type":"ContainerStarted","Data":"8a6868998cfd7386f4464cfd168e9a7987c55091b4db05308fe46c7bb2f3c235"} Sep 16 01:17:49.845857 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:49.845826 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" event={"ID":"ad694c3c-8319-4a83-a520-a6a13c1a077e","Type":"ContainerStarted","Data":"071bcf5e48ef88daf9d3272113c724aadb2962b5474d99df843fb8261cde90ae"} Sep 16 01:17:49.864698 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:49.864651 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" podStartSLOduration=1.472819209 podStartE2EDuration="4.86463796s" totalImagesPullingTime="3.391818751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:49.862713121 +0000 UTC m=+100.115135723" watchObservedRunningTime="2026-09-16 01:17:49.86463796 +0000 UTC m=+100.117060549" Sep 16 01:17:49.877611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:49.877572 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" podStartSLOduration=1.867280659 podStartE2EDuration="3.87755853s" totalImagesPullingTime="2.010277871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:49.876748003 +0000 UTC m=+100.129170605" watchObservedRunningTime="2026-09-16 01:17:49.87755853 +0000 UTC m=+100.129981118" Sep 16 01:17:50.578371 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:50.578337 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:51.128105 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:51.128069 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:51.228644 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:51.228613 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:53.107504 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:53.107470 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:53.207732 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:53.207701 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:53.710538 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:53.710510 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q6xx7" Sep 16 01:17:55.855388 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:55.855362 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7b4c7d77c7-wjvsp" Sep 16 01:17:57.149039 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:57.149003 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:57.149456 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:57.149108 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:57.153870 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:57.153847 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:57.883105 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:57.883079 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-887c86869-ftklv" Sep 16 01:17:57.924101 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:57.924075 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77d9cf858b-6jstg"] Sep 16 01:17:58.022148 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:58.022082 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:58.122833 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:58.122801 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:58.179295 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:58.179232 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:58.434600 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:58.434536 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:58.535890 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:58.535865 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:59.823418 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:59.823379 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:17:59.874658 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:17:59.874629 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:00.229776 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:00.229735 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:00.329613 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:00.329571 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:01.725013 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:01.724967 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:01.825521 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:01.825492 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:02.881430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:02.881398 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:02.982751 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:02.982722 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:06.663850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:06.663816 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:18:06.664254 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:06.663865 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:18:08.921088 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:08.921059 2558 generic.go:356] "Generic (PLEG): container finished" podID="d157e624-20f1-4d08-a7aa-23a3b0b1d190" containerID="2a7e8f66015d6d1bc9c2c6492b738bc89e21a9079df5246f1a0f0178087835a2" exitCode=0 Sep 16 01:18:08.921436 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:08.921126 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dgbrm" event={"ID":"d157e624-20f1-4d08-a7aa-23a3b0b1d190","Type":"ContainerDied","Data":"2a7e8f66015d6d1bc9c2c6492b738bc89e21a9079df5246f1a0f0178087835a2"} Sep 16 01:18:08.921477 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:08.921447 2558 scope.go:117] "RemoveContainer" containerID="2a7e8f66015d6d1bc9c2c6492b738bc89e21a9079df5246f1a0f0178087835a2" Sep 16 01:18:09.926817 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:09.926780 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dgbrm" event={"ID":"d157e624-20f1-4d08-a7aa-23a3b0b1d190","Type":"ContainerStarted","Data":"88e0ecc073435d6e67f74402b667c54cde11f98ba4c86ac7d7855409c29aae6a"} Sep 16 01:18:12.079347 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:12.079316 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-gcbb4_e7b745db-d94b-428e-8012-cab5a3d44d1d/kube-state-metrics/0.log" Sep 16 01:18:12.281171 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:12.281134 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-gcbb4_e7b745db-d94b-428e-8012-cab5a3d44d1d/kube-rbac-proxy-main/0.log" Sep 16 01:18:12.480034 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:12.480010 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-gcbb4_e7b745db-d94b-428e-8012-cab5a3d44d1d/kube-rbac-proxy-self/0.log" Sep 16 01:18:12.678170 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:12.678133 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6496c56f47-t8xsm_ad694c3c-8319-4a83-a520-a6a13c1a077e/metrics-server/0.log" Sep 16 01:18:13.438717 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:13.438683 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:13.492816 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:13.492789 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:13.678083 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:13.678062 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh6bv_af3a00ef-4f24-41c6-8ec5-65f9fc8685ac/init-textfile/0.log" Sep 16 01:18:13.880649 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:13.880626 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh6bv_af3a00ef-4f24-41c6-8ec5-65f9fc8685ac/node-exporter/0.log" Sep 16 01:18:14.078487 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:14.078457 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh6bv_af3a00ef-4f24-41c6-8ec5-65f9fc8685ac/kube-rbac-proxy/0.log" Sep 16 01:18:17.279273 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:17.279243 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-x9kmd_2ca52899-0cc2-4d38-942f-3659d1113017/prometheus-operator-admission-webhook/0.log" Sep 16 01:18:17.479252 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:17.479221 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4c7d77c7-wjvsp_a734a692-d38a-4b2b-b932-de84fea3dceb/thanos-query/0.log" Sep 16 01:18:17.678984 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:17.678961 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4c7d77c7-wjvsp_a734a692-d38a-4b2b-b932-de84fea3dceb/kube-rbac-proxy-web/0.log" Sep 16 01:18:17.878445 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:17.878412 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4c7d77c7-wjvsp_a734a692-d38a-4b2b-b932-de84fea3dceb/kube-rbac-proxy/0.log" Sep 16 01:18:18.078391 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:18.078324 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4c7d77c7-wjvsp_a734a692-d38a-4b2b-b932-de84fea3dceb/prom-label-proxy/0.log" Sep 16 01:18:18.278604 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:18.278580 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4c7d77c7-wjvsp_a734a692-d38a-4b2b-b932-de84fea3dceb/kube-rbac-proxy-rules/0.log" Sep 16 01:18:18.478572 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:18.478545 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4c7d77c7-wjvsp_a734a692-d38a-4b2b-b932-de84fea3dceb/kube-rbac-proxy-metrics/0.log" Sep 16 01:18:18.878695 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:18.878664 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:18:19.080954 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:19.080919 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/2.log" Sep 16 01:18:19.279234 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:19.279174 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77d9cf858b-6jstg_aee7b430-f393-464c-bb00-4e978de04450/console/0.log" Sep 16 01:18:19.481192 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:19.481167 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-887c86869-ftklv_144e3b20-3f4b-45f6-ac63-31371e640363/console/0.log" Sep 16 01:18:20.479357 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:20.479329 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zczs4_0974aa23-1b99-46bd-a1b0-fdc7f7657069/serve-healthcheck-canary/0.log" Sep 16 01:18:22.950694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:22.950648 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77d9cf858b-6jstg" podUID="aee7b430-f393-464c-bb00-4e978de04450" containerName="console" containerID="cri-o://6ee6393b8c446a67b33cbd7b0a05358621f1b5ed1dc08bdf3722260ee39b4ac2" gracePeriod=15 Sep 16 01:18:23.222716 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.222330 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:18:23.281188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.279415 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aee7b430-f393-464c-bb00-4e978de04450-console-serving-cert\") pod \"aee7b430-f393-464c-bb00-4e978de04450\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " Sep 16 01:18:23.281188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.279465 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-service-ca\") pod \"aee7b430-f393-464c-bb00-4e978de04450\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " Sep 16 01:18:23.281188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.279513 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aee7b430-f393-464c-bb00-4e978de04450-console-oauth-config\") pod \"aee7b430-f393-464c-bb00-4e978de04450\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " Sep 16 01:18:23.281188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.279557 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-oauth-serving-cert\") pod \"aee7b430-f393-464c-bb00-4e978de04450\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " Sep 16 01:18:23.281188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.279605 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6bvb\" (UniqueName: \"kubernetes.io/projected/aee7b430-f393-464c-bb00-4e978de04450-kube-api-access-j6bvb\") pod \"aee7b430-f393-464c-bb00-4e978de04450\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " Sep 16 01:18:23.281188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.279688 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-console-config\") pod \"aee7b430-f393-464c-bb00-4e978de04450\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " Sep 16 01:18:23.281188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.279716 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-trusted-ca-bundle\") pod \"aee7b430-f393-464c-bb00-4e978de04450\" (UID: \"aee7b430-f393-464c-bb00-4e978de04450\") " Sep 16 01:18:23.281188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.280339 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aee7b430-f393-464c-bb00-4e978de04450" (UID: "aee7b430-f393-464c-bb00-4e978de04450"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:18:23.282716 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.282683 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-service-ca" (OuterVolumeSpecName: "service-ca") pod "aee7b430-f393-464c-bb00-4e978de04450" (UID: "aee7b430-f393-464c-bb00-4e978de04450"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:18:23.282716 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.282698 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-console-config" (OuterVolumeSpecName: "console-config") pod "aee7b430-f393-464c-bb00-4e978de04450" (UID: "aee7b430-f393-464c-bb00-4e978de04450"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:18:23.282890 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.282762 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aee7b430-f393-464c-bb00-4e978de04450" (UID: "aee7b430-f393-464c-bb00-4e978de04450"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:18:23.283911 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.283886 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee7b430-f393-464c-bb00-4e978de04450-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aee7b430-f393-464c-bb00-4e978de04450" (UID: "aee7b430-f393-464c-bb00-4e978de04450"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:18:23.284611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.284579 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee7b430-f393-464c-bb00-4e978de04450-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aee7b430-f393-464c-bb00-4e978de04450" (UID: "aee7b430-f393-464c-bb00-4e978de04450"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:18:23.285208 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.285143 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee7b430-f393-464c-bb00-4e978de04450-kube-api-access-j6bvb" (OuterVolumeSpecName: "kube-api-access-j6bvb") pod "aee7b430-f393-464c-bb00-4e978de04450" (UID: "aee7b430-f393-464c-bb00-4e978de04450"). InnerVolumeSpecName "kube-api-access-j6bvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:18:23.380821 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.380791 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-console-config\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:18:23.380821 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.380820 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-trusted-ca-bundle\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:18:23.380821 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.380830 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aee7b430-f393-464c-bb00-4e978de04450-console-serving-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:18:23.381040 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.380838 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-service-ca\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:18:23.381040 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.380847 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aee7b430-f393-464c-bb00-4e978de04450-console-oauth-config\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:18:23.381040 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.380862 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aee7b430-f393-464c-bb00-4e978de04450-oauth-serving-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:18:23.381040 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.380871 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6bvb\" (UniqueName: \"kubernetes.io/projected/aee7b430-f393-464c-bb00-4e978de04450-kube-api-access-j6bvb\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:18:23.985543 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.985510 2558 generic.go:356] "Generic (PLEG): container finished" podID="aee7b430-f393-464c-bb00-4e978de04450" containerID="6ee6393b8c446a67b33cbd7b0a05358621f1b5ed1dc08bdf3722260ee39b4ac2" exitCode=0 Sep 16 01:18:23.985543 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.985550 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d9cf858b-6jstg" event={"ID":"aee7b430-f393-464c-bb00-4e978de04450","Type":"ContainerDied","Data":"6ee6393b8c446a67b33cbd7b0a05358621f1b5ed1dc08bdf3722260ee39b4ac2"} Sep 16 01:18:23.986049 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.985556 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d9cf858b-6jstg" Sep 16 01:18:23.986049 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.985591 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d9cf858b-6jstg" event={"ID":"aee7b430-f393-464c-bb00-4e978de04450","Type":"ContainerDied","Data":"205f86fc22a9f1cd6ea5c95cee3b88d5de6b586708c4f5ef78afa10b5691b7a1"} Sep 16 01:18:23.986049 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.985611 2558 scope.go:117] "RemoveContainer" containerID="6ee6393b8c446a67b33cbd7b0a05358621f1b5ed1dc08bdf3722260ee39b4ac2" Sep 16 01:18:23.995456 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.995436 2558 scope.go:117] "RemoveContainer" containerID="6ee6393b8c446a67b33cbd7b0a05358621f1b5ed1dc08bdf3722260ee39b4ac2" Sep 16 01:18:23.995772 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:18:23.995743 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee6393b8c446a67b33cbd7b0a05358621f1b5ed1dc08bdf3722260ee39b4ac2\": container with ID starting with 6ee6393b8c446a67b33cbd7b0a05358621f1b5ed1dc08bdf3722260ee39b4ac2 not found: ID does not exist" containerID="6ee6393b8c446a67b33cbd7b0a05358621f1b5ed1dc08bdf3722260ee39b4ac2" Sep 16 01:18:23.995831 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:23.995785 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee6393b8c446a67b33cbd7b0a05358621f1b5ed1dc08bdf3722260ee39b4ac2"} err="failed to get container status \"6ee6393b8c446a67b33cbd7b0a05358621f1b5ed1dc08bdf3722260ee39b4ac2\": rpc error: code = NotFound desc = could not find container \"6ee6393b8c446a67b33cbd7b0a05358621f1b5ed1dc08bdf3722260ee39b4ac2\": container with ID starting with 6ee6393b8c446a67b33cbd7b0a05358621f1b5ed1dc08bdf3722260ee39b4ac2 not found: ID does not exist" Sep 16 01:18:24.007738 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:24.007690 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77d9cf858b-6jstg"] Sep 16 01:18:24.009664 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:24.009637 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77d9cf858b-6jstg"] Sep 16 01:18:24.293957 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:24.293886 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aee7b430-f393-464c-bb00-4e978de04450" path="/var/lib/kubelet/pods/aee7b430-f393-464c-bb00-4e978de04450/volumes" Sep 16 01:18:26.669760 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:26.669734 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:18:26.673527 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:26.673503 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6496c56f47-t8xsm" Sep 16 01:18:33.856905 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:33.856864 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:47.477979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:47.477945 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:47.579674 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:47.579641 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:47.679654 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:47.679623 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:58.441250 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:58.441217 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:18:58.540612 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:18:58.540587 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:01.265095 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:01.265058 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:01.466306 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:01.466243 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:08.987725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:08.987691 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8c5b8fb5f-cknt9"] Sep 16 01:19:08.988080 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:08.987985 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee7b430-f393-464c-bb00-4e978de04450" containerName="console" Sep 16 01:19:08.988080 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:08.987996 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee7b430-f393-464c-bb00-4e978de04450" containerName="console" Sep 16 01:19:08.988080 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:08.988065 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee7b430-f393-464c-bb00-4e978de04450" containerName="console" Sep 16 01:19:08.990993 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:08.990962 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.002908 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.002877 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8c5b8fb5f-cknt9"] Sep 16 01:19:09.034787 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.034757 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:09.085005 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.084977 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-service-ca\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.085104 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.085009 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-oauth-config\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.085104 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.085027 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-trusted-ca-bundle\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.085104 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.085050 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-oauth-serving-cert\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.085231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.085130 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-config\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.085231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.085187 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-serving-cert\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.085231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.085207 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmzwn\" (UniqueName: \"kubernetes.io/projected/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-kube-api-access-rmzwn\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.088800 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.088780 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:09.185731 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.185709 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-oauth-serving-cert\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.185840 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.185746 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-config\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.185840 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.185779 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-serving-cert\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.185840 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.185803 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmzwn\" (UniqueName: \"kubernetes.io/projected/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-kube-api-access-rmzwn\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.185965 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.185850 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-service-ca\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.185965 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.185877 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-oauth-config\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.185965 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.185898 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-trusted-ca-bundle\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.186538 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.186514 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-config\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.186635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.186568 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-oauth-serving-cert\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.186635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.186616 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-service-ca\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.186753 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.186736 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-trusted-ca-bundle\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.189148 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.189123 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-oauth-config\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.189337 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.189215 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-serving-cert\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.193134 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.193116 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmzwn\" (UniqueName: \"kubernetes.io/projected/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-kube-api-access-rmzwn\") pod \"console-8c5b8fb5f-cknt9\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.306751 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.306690 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:09.577410 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:09.577364 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8c5b8fb5f-cknt9"] Sep 16 01:19:09.580052 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:19:09.580022 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c7fcb7_2dee_4fb5_a9df_b59f3b3947c3.slice/crio-99c4ff12d1eae7a95142926d3ba08c0e42d7f8ab628ec9d6d437cfdebe109bd8 WatchSource:0}: Error finding container 99c4ff12d1eae7a95142926d3ba08c0e42d7f8ab628ec9d6d437cfdebe109bd8: Status 404 returned error can't find the container with id 99c4ff12d1eae7a95142926d3ba08c0e42d7f8ab628ec9d6d437cfdebe109bd8 Sep 16 01:19:10.145077 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:10.145044 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c5b8fb5f-cknt9" event={"ID":"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3","Type":"ContainerStarted","Data":"1bfefccea885a6f47a08ecffa93632f89d29539ad5e9bcb11b5d90298c0486c3"} Sep 16 01:19:10.145443 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:10.145114 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c5b8fb5f-cknt9" event={"ID":"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3","Type":"ContainerStarted","Data":"99c4ff12d1eae7a95142926d3ba08c0e42d7f8ab628ec9d6d437cfdebe109bd8"} Sep 16 01:19:10.161718 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:10.161675 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8c5b8fb5f-cknt9" podStartSLOduration=2.161660238 podStartE2EDuration="2.161660238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:19:10.1606823 +0000 UTC m=+180.413104926" watchObservedRunningTime="2026-09-16 01:19:10.161660238 +0000 UTC m=+180.414082826" Sep 16 01:19:16.435587 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:16.435551 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:16.491054 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:16.491022 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:19.307479 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:19.307450 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:19.307862 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:19.307493 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:19.312078 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:19.312055 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:20.187486 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.187461 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:20.228075 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.228046 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-887c86869-ftklv"] Sep 16 01:19:20.296756 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.296716 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67b955f76d-5566h"] Sep 16 01:19:20.304270 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.304246 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.307949 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.307913 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67b955f76d-5566h"] Sep 16 01:19:20.366405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.366377 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13bba4d5-c971-40af-b470-52b17914a115-console-oauth-config\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.366572 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.366412 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-console-config\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.366572 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.366515 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-trusted-ca-bundle\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.366572 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.366556 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-service-ca\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.366746 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.366630 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13bba4d5-c971-40af-b470-52b17914a115-console-serving-cert\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.366746 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.366693 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns5tz\" (UniqueName: \"kubernetes.io/projected/13bba4d5-c971-40af-b470-52b17914a115-kube-api-access-ns5tz\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.366746 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.366739 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-oauth-serving-cert\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.467278 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.467216 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-trusted-ca-bundle\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.467278 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.467244 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-service-ca\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.467278 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.467267 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13bba4d5-c971-40af-b470-52b17914a115-console-serving-cert\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.467478 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.467286 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns5tz\" (UniqueName: \"kubernetes.io/projected/13bba4d5-c971-40af-b470-52b17914a115-kube-api-access-ns5tz\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.467478 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.467318 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-oauth-serving-cert\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.467478 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.467366 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13bba4d5-c971-40af-b470-52b17914a115-console-oauth-config\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.467478 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.467397 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-console-config\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.468088 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.468062 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-service-ca\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.468229 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.468088 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-trusted-ca-bundle\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.468229 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.468098 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-console-config\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.468331 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.468311 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-oauth-serving-cert\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.469860 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.469827 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13bba4d5-c971-40af-b470-52b17914a115-console-oauth-config\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.470042 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.470020 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13bba4d5-c971-40af-b470-52b17914a115-console-serving-cert\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.475393 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.475373 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns5tz\" (UniqueName: \"kubernetes.io/projected/13bba4d5-c971-40af-b470-52b17914a115-kube-api-access-ns5tz\") pod \"console-67b955f76d-5566h\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.615655 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.615625 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:20.676040 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.675910 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:20.889258 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:20.889234 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67b955f76d-5566h"] Sep 16 01:19:20.891714 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:19:20.891681 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bba4d5_c971_40af_b470_52b17914a115.slice/crio-4ef31d7f26be6dea95f716faad5e68e0281b8f2341e373c4d2ce3142b15c4410 WatchSource:0}: Error finding container 4ef31d7f26be6dea95f716faad5e68e0281b8f2341e373c4d2ce3142b15c4410: Status 404 returned error can't find the container with id 4ef31d7f26be6dea95f716faad5e68e0281b8f2341e373c4d2ce3142b15c4410 Sep 16 01:19:21.191334 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:21.191248 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b955f76d-5566h" event={"ID":"13bba4d5-c971-40af-b470-52b17914a115","Type":"ContainerStarted","Data":"1c0dfc7799ad9b307166a96419f4a42b930b065dc854ff8c18c273a2666f7fbd"} Sep 16 01:19:21.191334 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:21.191299 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b955f76d-5566h" event={"ID":"13bba4d5-c971-40af-b470-52b17914a115","Type":"ContainerStarted","Data":"4ef31d7f26be6dea95f716faad5e68e0281b8f2341e373c4d2ce3142b15c4410"} Sep 16 01:19:21.208147 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:21.208104 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67b955f76d-5566h" podStartSLOduration=1.208089043 podStartE2EDuration="1.208089043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:19:21.206770896 +0000 UTC m=+191.459193510" watchObservedRunningTime="2026-09-16 01:19:21.208089043 +0000 UTC m=+191.460511697" Sep 16 01:19:21.274597 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:21.274570 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:21.375009 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:21.374970 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:28.334620 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:28.334586 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:28.383406 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:28.383377 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:30.616247 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:30.616215 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:30.616611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:30.616277 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:30.620602 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:30.620581 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:31.234420 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:31.234394 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:19:31.276883 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:31.276852 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8c5b8fb5f-cknt9"] Sep 16 01:19:31.365002 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:31.364922 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:31.419775 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:31.419743 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:31.470950 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:31.470918 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:39.582210 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:39.582179 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:39.683231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:39.683200 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:43.443352 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:43.443315 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:43.543443 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:43.543415 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:45.250143 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.250090 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-887c86869-ftklv" podUID="144e3b20-3f4b-45f6-ac63-31371e640363" containerName="console" containerID="cri-o://4a1d41e8dd2ffa93a3fcd0e4eb1c2cd2cd1e75771700f009d7a0115442c90ca2" gracePeriod=15 Sep 16 01:19:45.492085 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.492062 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-887c86869-ftklv" Sep 16 01:19:45.638578 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.638549 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cpml\" (UniqueName: \"kubernetes.io/projected/144e3b20-3f4b-45f6-ac63-31371e640363-kube-api-access-7cpml\") pod \"144e3b20-3f4b-45f6-ac63-31371e640363\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " Sep 16 01:19:45.638773 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.638600 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-service-ca\") pod \"144e3b20-3f4b-45f6-ac63-31371e640363\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " Sep 16 01:19:45.638773 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.638621 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-trusted-ca-bundle\") pod \"144e3b20-3f4b-45f6-ac63-31371e640363\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " Sep 16 01:19:45.638773 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.638650 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-console-config\") pod \"144e3b20-3f4b-45f6-ac63-31371e640363\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " Sep 16 01:19:45.638773 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.638681 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/144e3b20-3f4b-45f6-ac63-31371e640363-console-oauth-config\") pod \"144e3b20-3f4b-45f6-ac63-31371e640363\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " Sep 16 01:19:45.638773 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.638718 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-oauth-serving-cert\") pod \"144e3b20-3f4b-45f6-ac63-31371e640363\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " Sep 16 01:19:45.639021 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.638779 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/144e3b20-3f4b-45f6-ac63-31371e640363-console-serving-cert\") pod \"144e3b20-3f4b-45f6-ac63-31371e640363\" (UID: \"144e3b20-3f4b-45f6-ac63-31371e640363\") " Sep 16 01:19:45.639074 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.639035 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-console-config" (OuterVolumeSpecName: "console-config") pod "144e3b20-3f4b-45f6-ac63-31371e640363" (UID: "144e3b20-3f4b-45f6-ac63-31371e640363"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:19:45.639130 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.639096 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "144e3b20-3f4b-45f6-ac63-31371e640363" (UID: "144e3b20-3f4b-45f6-ac63-31371e640363"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:19:45.639130 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.639117 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-service-ca" (OuterVolumeSpecName: "service-ca") pod "144e3b20-3f4b-45f6-ac63-31371e640363" (UID: "144e3b20-3f4b-45f6-ac63-31371e640363"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:19:45.639350 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.639327 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "144e3b20-3f4b-45f6-ac63-31371e640363" (UID: "144e3b20-3f4b-45f6-ac63-31371e640363"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:19:45.641091 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.641053 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144e3b20-3f4b-45f6-ac63-31371e640363-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "144e3b20-3f4b-45f6-ac63-31371e640363" (UID: "144e3b20-3f4b-45f6-ac63-31371e640363"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:19:45.641429 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.641401 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144e3b20-3f4b-45f6-ac63-31371e640363-kube-api-access-7cpml" (OuterVolumeSpecName: "kube-api-access-7cpml") pod "144e3b20-3f4b-45f6-ac63-31371e640363" (UID: "144e3b20-3f4b-45f6-ac63-31371e640363"). InnerVolumeSpecName "kube-api-access-7cpml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:19:45.641596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.641568 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144e3b20-3f4b-45f6-ac63-31371e640363-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "144e3b20-3f4b-45f6-ac63-31371e640363" (UID: "144e3b20-3f4b-45f6-ac63-31371e640363"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:19:45.739801 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.739770 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-oauth-serving-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:45.739801 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.739796 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/144e3b20-3f4b-45f6-ac63-31371e640363-console-serving-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:45.739951 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.739807 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cpml\" (UniqueName: \"kubernetes.io/projected/144e3b20-3f4b-45f6-ac63-31371e640363-kube-api-access-7cpml\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:45.739951 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.739817 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-service-ca\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:45.739951 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.739825 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-trusted-ca-bundle\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:45.739951 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.739833 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/144e3b20-3f4b-45f6-ac63-31371e640363-console-config\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:45.739951 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:45.739841 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/144e3b20-3f4b-45f6-ac63-31371e640363-console-oauth-config\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:46.286270 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:46.286241 2558 generic.go:356] "Generic (PLEG): container finished" podID="144e3b20-3f4b-45f6-ac63-31371e640363" containerID="4a1d41e8dd2ffa93a3fcd0e4eb1c2cd2cd1e75771700f009d7a0115442c90ca2" exitCode=0 Sep 16 01:19:46.286694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:46.286285 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-887c86869-ftklv" Sep 16 01:19:46.286694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:46.286291 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-887c86869-ftklv" event={"ID":"144e3b20-3f4b-45f6-ac63-31371e640363","Type":"ContainerDied","Data":"4a1d41e8dd2ffa93a3fcd0e4eb1c2cd2cd1e75771700f009d7a0115442c90ca2"} Sep 16 01:19:46.286694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:46.286328 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-887c86869-ftklv" event={"ID":"144e3b20-3f4b-45f6-ac63-31371e640363","Type":"ContainerDied","Data":"83e8f8d818603ae43514d5c750549c6ca76715914212cf6bbe3c988cc32f9d96"} Sep 16 01:19:46.286694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:46.286347 2558 scope.go:117] "RemoveContainer" containerID="4a1d41e8dd2ffa93a3fcd0e4eb1c2cd2cd1e75771700f009d7a0115442c90ca2" Sep 16 01:19:46.295452 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:46.295433 2558 scope.go:117] "RemoveContainer" containerID="4a1d41e8dd2ffa93a3fcd0e4eb1c2cd2cd1e75771700f009d7a0115442c90ca2" Sep 16 01:19:46.295746 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:19:46.295728 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1d41e8dd2ffa93a3fcd0e4eb1c2cd2cd1e75771700f009d7a0115442c90ca2\": container with ID starting with 4a1d41e8dd2ffa93a3fcd0e4eb1c2cd2cd1e75771700f009d7a0115442c90ca2 not found: ID does not exist" containerID="4a1d41e8dd2ffa93a3fcd0e4eb1c2cd2cd1e75771700f009d7a0115442c90ca2" Sep 16 01:19:46.295796 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:46.295755 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1d41e8dd2ffa93a3fcd0e4eb1c2cd2cd1e75771700f009d7a0115442c90ca2"} err="failed to get container status \"4a1d41e8dd2ffa93a3fcd0e4eb1c2cd2cd1e75771700f009d7a0115442c90ca2\": rpc error: code = NotFound desc = could not find container \"4a1d41e8dd2ffa93a3fcd0e4eb1c2cd2cd1e75771700f009d7a0115442c90ca2\": container with ID starting with 4a1d41e8dd2ffa93a3fcd0e4eb1c2cd2cd1e75771700f009d7a0115442c90ca2 not found: ID does not exist" Sep 16 01:19:46.307643 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:46.307621 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-887c86869-ftklv"] Sep 16 01:19:46.313074 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:46.313052 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-887c86869-ftklv"] Sep 16 01:19:48.293290 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:48.293251 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144e3b20-3f4b-45f6-ac63-31371e640363" path="/var/lib/kubelet/pods/144e3b20-3f4b-45f6-ac63-31371e640363/volumes" Sep 16 01:19:53.405405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:53.405360 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:19:56.297119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.297070 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8c5b8fb5f-cknt9" podUID="b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3" containerName="console" containerID="cri-o://1bfefccea885a6f47a08ecffa93632f89d29539ad5e9bcb11b5d90298c0486c3" gracePeriod=15 Sep 16 01:19:56.543631 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.543608 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:56.714697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.714666 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-service-ca\") pod \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " Sep 16 01:19:56.714849 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.714703 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmzwn\" (UniqueName: \"kubernetes.io/projected/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-kube-api-access-rmzwn\") pod \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " Sep 16 01:19:56.714849 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.714754 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-serving-cert\") pod \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " Sep 16 01:19:56.714849 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.714810 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-trusted-ca-bundle\") pod \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " Sep 16 01:19:56.714849 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.714843 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-oauth-serving-cert\") pod \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " Sep 16 01:19:56.715119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.714872 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-config\") pod \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " Sep 16 01:19:56.715119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.714923 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-oauth-config\") pod \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\" (UID: \"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3\") " Sep 16 01:19:56.715119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.715072 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-service-ca" (OuterVolumeSpecName: "service-ca") pod "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3" (UID: "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:19:56.715315 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.715272 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-service-ca\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:56.715315 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.715282 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3" (UID: "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:19:56.715412 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.715319 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3" (UID: "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:19:56.715469 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.715448 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-config" (OuterVolumeSpecName: "console-config") pod "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3" (UID: "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:19:56.717078 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.717054 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3" (UID: "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:19:56.717078 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.717059 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-kube-api-access-rmzwn" (OuterVolumeSpecName: "kube-api-access-rmzwn") pod "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3" (UID: "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3"). InnerVolumeSpecName "kube-api-access-rmzwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:19:56.717236 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.717103 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3" (UID: "b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:19:56.816237 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.816213 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmzwn\" (UniqueName: \"kubernetes.io/projected/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-kube-api-access-rmzwn\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:56.816237 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.816233 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-serving-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:56.816357 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.816242 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-trusted-ca-bundle\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:56.816357 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.816250 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-oauth-serving-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:56.816357 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.816260 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-config\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:56.816357 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:56.816268 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3-console-oauth-config\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:19:57.325630 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:57.325602 2558 generic.go:356] "Generic (PLEG): container finished" podID="b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3" containerID="1bfefccea885a6f47a08ecffa93632f89d29539ad5e9bcb11b5d90298c0486c3" exitCode=0 Sep 16 01:19:57.326078 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:57.325645 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c5b8fb5f-cknt9" Sep 16 01:19:57.326078 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:57.325688 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c5b8fb5f-cknt9" event={"ID":"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3","Type":"ContainerDied","Data":"1bfefccea885a6f47a08ecffa93632f89d29539ad5e9bcb11b5d90298c0486c3"} Sep 16 01:19:57.326078 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:57.325743 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c5b8fb5f-cknt9" event={"ID":"b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3","Type":"ContainerDied","Data":"99c4ff12d1eae7a95142926d3ba08c0e42d7f8ab628ec9d6d437cfdebe109bd8"} Sep 16 01:19:57.326078 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:57.325767 2558 scope.go:117] "RemoveContainer" containerID="1bfefccea885a6f47a08ecffa93632f89d29539ad5e9bcb11b5d90298c0486c3" Sep 16 01:19:57.334534 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:57.334519 2558 scope.go:117] "RemoveContainer" containerID="1bfefccea885a6f47a08ecffa93632f89d29539ad5e9bcb11b5d90298c0486c3" Sep 16 01:19:57.334808 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:19:57.334790 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfefccea885a6f47a08ecffa93632f89d29539ad5e9bcb11b5d90298c0486c3\": container with ID starting with 1bfefccea885a6f47a08ecffa93632f89d29539ad5e9bcb11b5d90298c0486c3 not found: ID does not exist" containerID="1bfefccea885a6f47a08ecffa93632f89d29539ad5e9bcb11b5d90298c0486c3" Sep 16 01:19:57.334854 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:57.334816 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfefccea885a6f47a08ecffa93632f89d29539ad5e9bcb11b5d90298c0486c3"} err="failed to get container status \"1bfefccea885a6f47a08ecffa93632f89d29539ad5e9bcb11b5d90298c0486c3\": rpc error: code = NotFound desc = could not find container \"1bfefccea885a6f47a08ecffa93632f89d29539ad5e9bcb11b5d90298c0486c3\": container with ID starting with 1bfefccea885a6f47a08ecffa93632f89d29539ad5e9bcb11b5d90298c0486c3 not found: ID does not exist" Sep 16 01:19:57.345913 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:57.345889 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8c5b8fb5f-cknt9"] Sep 16 01:19:57.349209 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:57.349187 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8c5b8fb5f-cknt9"] Sep 16 01:19:58.293290 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:19:58.293260 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3" path="/var/lib/kubelet/pods/b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3/volumes" Sep 16 01:20:14.830700 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:14.830664 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:14.925936 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:14.925908 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:15.473915 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.473886 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49"] Sep 16 01:20:15.474205 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.474193 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3" containerName="console" Sep 16 01:20:15.474258 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.474212 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3" containerName="console" Sep 16 01:20:15.474258 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.474230 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144e3b20-3f4b-45f6-ac63-31371e640363" containerName="console" Sep 16 01:20:15.474258 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.474235 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="144e3b20-3f4b-45f6-ac63-31371e640363" containerName="console" Sep 16 01:20:15.474396 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.474288 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="144e3b20-3f4b-45f6-ac63-31371e640363" containerName="console" Sep 16 01:20:15.474396 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.474296 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c7fcb7-2dee-4fb5-a9df-b59f3b3947c3" containerName="console" Sep 16 01:20:15.479259 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.479221 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" Sep 16 01:20:15.481966 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.481947 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hgbd4\"" Sep 16 01:20:15.482298 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.482281 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 01:20:15.482634 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.482617 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 01:20:15.485597 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.485569 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49"] Sep 16 01:20:15.553412 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.553385 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e13b49d4-a070-4253-b560-7a3c81cea080-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49\" (UID: \"e13b49d4-a070-4253-b560-7a3c81cea080\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" Sep 16 01:20:15.553551 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.553425 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e13b49d4-a070-4253-b560-7a3c81cea080-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49\" (UID: \"e13b49d4-a070-4253-b560-7a3c81cea080\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" Sep 16 01:20:15.553551 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.553454 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7d85\" (UniqueName: \"kubernetes.io/projected/e13b49d4-a070-4253-b560-7a3c81cea080-kube-api-access-z7d85\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49\" (UID: \"e13b49d4-a070-4253-b560-7a3c81cea080\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" Sep 16 01:20:15.654431 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.654406 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7d85\" (UniqueName: \"kubernetes.io/projected/e13b49d4-a070-4253-b560-7a3c81cea080-kube-api-access-z7d85\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49\" (UID: \"e13b49d4-a070-4253-b560-7a3c81cea080\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" Sep 16 01:20:15.654542 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.654481 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e13b49d4-a070-4253-b560-7a3c81cea080-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49\" (UID: \"e13b49d4-a070-4253-b560-7a3c81cea080\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" Sep 16 01:20:15.654542 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.654526 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e13b49d4-a070-4253-b560-7a3c81cea080-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49\" (UID: \"e13b49d4-a070-4253-b560-7a3c81cea080\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" Sep 16 01:20:15.654891 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.654873 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e13b49d4-a070-4253-b560-7a3c81cea080-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49\" (UID: \"e13b49d4-a070-4253-b560-7a3c81cea080\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" Sep 16 01:20:15.654934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.654873 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e13b49d4-a070-4253-b560-7a3c81cea080-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49\" (UID: \"e13b49d4-a070-4253-b560-7a3c81cea080\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" Sep 16 01:20:15.661742 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.661720 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7d85\" (UniqueName: \"kubernetes.io/projected/e13b49d4-a070-4253-b560-7a3c81cea080-kube-api-access-z7d85\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49\" (UID: \"e13b49d4-a070-4253-b560-7a3c81cea080\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" Sep 16 01:20:15.790307 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:15.790244 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" Sep 16 01:20:16.058424 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:16.058367 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49"] Sep 16 01:20:16.060730 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:20:16.060706 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b49d4_a070_4253_b560_7a3c81cea080.slice/crio-a4c132d7d2a6a4b632790e08582fcc5b98101facdfcd7b36857ee37fd7516f7e WatchSource:0}: Error finding container a4c132d7d2a6a4b632790e08582fcc5b98101facdfcd7b36857ee37fd7516f7e: Status 404 returned error can't find the container with id a4c132d7d2a6a4b632790e08582fcc5b98101facdfcd7b36857ee37fd7516f7e Sep 16 01:20:16.394807 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:16.394774 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" event={"ID":"e13b49d4-a070-4253-b560-7a3c81cea080","Type":"ContainerStarted","Data":"a4c132d7d2a6a4b632790e08582fcc5b98101facdfcd7b36857ee37fd7516f7e"} Sep 16 01:20:23.427071 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:23.427036 2558 generic.go:356] "Generic (PLEG): container finished" podID="e13b49d4-a070-4253-b560-7a3c81cea080" containerID="a4dfa73decc47d0230aab56a8f2bb5b873fd8cb92a6824d7464a3d8d17e7419f" exitCode=0 Sep 16 01:20:23.427478 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:23.427125 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" event={"ID":"e13b49d4-a070-4253-b560-7a3c81cea080","Type":"ContainerDied","Data":"a4dfa73decc47d0230aab56a8f2bb5b873fd8cb92a6824d7464a3d8d17e7419f"} Sep 16 01:20:26.440672 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:26.440636 2558 generic.go:356] "Generic (PLEG): container finished" podID="e13b49d4-a070-4253-b560-7a3c81cea080" containerID="5883355d600f860a58f09e4657952ccd7ad1b81f7bf460cbb7bb220b62e94f89" exitCode=0 Sep 16 01:20:26.441029 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:26.440717 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" event={"ID":"e13b49d4-a070-4253-b560-7a3c81cea080","Type":"ContainerDied","Data":"5883355d600f860a58f09e4657952ccd7ad1b81f7bf460cbb7bb220b62e94f89"} Sep 16 01:20:33.471629 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:33.471547 2558 generic.go:356] "Generic (PLEG): container finished" podID="e13b49d4-a070-4253-b560-7a3c81cea080" containerID="5284156eac1cdc3b9c3e144b222e981aa45e6d89618790e1ce800b8a39ac0db9" exitCode=0 Sep 16 01:20:33.471629 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:33.471591 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" event={"ID":"e13b49d4-a070-4253-b560-7a3c81cea080","Type":"ContainerDied","Data":"5284156eac1cdc3b9c3e144b222e981aa45e6d89618790e1ce800b8a39ac0db9"} Sep 16 01:20:34.594044 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:34.594019 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" Sep 16 01:20:34.678429 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:34.678395 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e13b49d4-a070-4253-b560-7a3c81cea080-bundle\") pod \"e13b49d4-a070-4253-b560-7a3c81cea080\" (UID: \"e13b49d4-a070-4253-b560-7a3c81cea080\") " Sep 16 01:20:34.678576 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:34.678466 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7d85\" (UniqueName: \"kubernetes.io/projected/e13b49d4-a070-4253-b560-7a3c81cea080-kube-api-access-z7d85\") pod \"e13b49d4-a070-4253-b560-7a3c81cea080\" (UID: \"e13b49d4-a070-4253-b560-7a3c81cea080\") " Sep 16 01:20:34.678576 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:34.678520 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e13b49d4-a070-4253-b560-7a3c81cea080-util\") pod \"e13b49d4-a070-4253-b560-7a3c81cea080\" (UID: \"e13b49d4-a070-4253-b560-7a3c81cea080\") " Sep 16 01:20:34.679276 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:34.679251 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13b49d4-a070-4253-b560-7a3c81cea080-bundle" (OuterVolumeSpecName: "bundle") pod "e13b49d4-a070-4253-b560-7a3c81cea080" (UID: "e13b49d4-a070-4253-b560-7a3c81cea080"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:20:34.680632 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:34.680606 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13b49d4-a070-4253-b560-7a3c81cea080-kube-api-access-z7d85" (OuterVolumeSpecName: "kube-api-access-z7d85") pod "e13b49d4-a070-4253-b560-7a3c81cea080" (UID: "e13b49d4-a070-4253-b560-7a3c81cea080"). InnerVolumeSpecName "kube-api-access-z7d85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:20:34.683184 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:34.683146 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13b49d4-a070-4253-b560-7a3c81cea080-util" (OuterVolumeSpecName: "util") pod "e13b49d4-a070-4253-b560-7a3c81cea080" (UID: "e13b49d4-a070-4253-b560-7a3c81cea080"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:20:34.779401 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:34.779334 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e13b49d4-a070-4253-b560-7a3c81cea080-util\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:20:34.779401 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:34.779361 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e13b49d4-a070-4253-b560-7a3c81cea080-bundle\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:20:34.779401 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:34.779374 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7d85\" (UniqueName: \"kubernetes.io/projected/e13b49d4-a070-4253-b560-7a3c81cea080-kube-api-access-z7d85\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:20:35.481142 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:35.481115 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" Sep 16 01:20:35.481295 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:35.481139 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hdl49" event={"ID":"e13b49d4-a070-4253-b560-7a3c81cea080","Type":"ContainerDied","Data":"a4c132d7d2a6a4b632790e08582fcc5b98101facdfcd7b36857ee37fd7516f7e"} Sep 16 01:20:35.481295 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:35.481205 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4c132d7d2a6a4b632790e08582fcc5b98101facdfcd7b36857ee37fd7516f7e" Sep 16 01:20:41.496319 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:41.496281 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:42.286031 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:42.285991 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:43.410935 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.410899 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj"] Sep 16 01:20:43.411313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.411197 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13b49d4-a070-4253-b560-7a3c81cea080" containerName="util" Sep 16 01:20:43.411313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.411209 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13b49d4-a070-4253-b560-7a3c81cea080" containerName="util" Sep 16 01:20:43.411313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.411224 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13b49d4-a070-4253-b560-7a3c81cea080" containerName="pull" Sep 16 01:20:43.411313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.411231 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13b49d4-a070-4253-b560-7a3c81cea080" containerName="pull" Sep 16 01:20:43.411313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.411240 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13b49d4-a070-4253-b560-7a3c81cea080" containerName="extract" Sep 16 01:20:43.411313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.411245 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13b49d4-a070-4253-b560-7a3c81cea080" containerName="extract" Sep 16 01:20:43.411313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.411303 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e13b49d4-a070-4253-b560-7a3c81cea080" containerName="extract" Sep 16 01:20:43.414474 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.414455 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj" Sep 16 01:20:43.416699 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.416678 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 16 01:20:43.416810 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.416796 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 16 01:20:43.416947 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.416931 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ddsnp\"" Sep 16 01:20:43.426518 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.426494 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj"] Sep 16 01:20:43.441210 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.441190 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21d2c26b-e753-4983-9fb9-223b8a4acf32-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hzmtj\" (UID: \"21d2c26b-e753-4983-9fb9-223b8a4acf32\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj" Sep 16 01:20:43.441320 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.441219 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj7zj\" (UniqueName: \"kubernetes.io/projected/21d2c26b-e753-4983-9fb9-223b8a4acf32-kube-api-access-cj7zj\") pod \"cert-manager-operator-controller-manager-55c444bf86-hzmtj\" (UID: \"21d2c26b-e753-4983-9fb9-223b8a4acf32\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj" Sep 16 01:20:43.541558 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.541535 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21d2c26b-e753-4983-9fb9-223b8a4acf32-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hzmtj\" (UID: \"21d2c26b-e753-4983-9fb9-223b8a4acf32\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj" Sep 16 01:20:43.541668 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.541562 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj7zj\" (UniqueName: \"kubernetes.io/projected/21d2c26b-e753-4983-9fb9-223b8a4acf32-kube-api-access-cj7zj\") pod \"cert-manager-operator-controller-manager-55c444bf86-hzmtj\" (UID: \"21d2c26b-e753-4983-9fb9-223b8a4acf32\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj" Sep 16 01:20:43.541898 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.541880 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21d2c26b-e753-4983-9fb9-223b8a4acf32-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hzmtj\" (UID: \"21d2c26b-e753-4983-9fb9-223b8a4acf32\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj" Sep 16 01:20:43.549611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.549589 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj7zj\" (UniqueName: \"kubernetes.io/projected/21d2c26b-e753-4983-9fb9-223b8a4acf32-kube-api-access-cj7zj\") pod \"cert-manager-operator-controller-manager-55c444bf86-hzmtj\" (UID: \"21d2c26b-e753-4983-9fb9-223b8a4acf32\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj" Sep 16 01:20:43.723667 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.723611 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj" Sep 16 01:20:43.997616 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:43.997591 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj"] Sep 16 01:20:43.999799 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:20:43.999774 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d2c26b_e753_4983_9fb9_223b8a4acf32.slice/crio-fb1680cea54ece60b573eb0b0b565028e52da352a0606f3d19dd8444c133226b WatchSource:0}: Error finding container fb1680cea54ece60b573eb0b0b565028e52da352a0606f3d19dd8444c133226b: Status 404 returned error can't find the container with id fb1680cea54ece60b573eb0b0b565028e52da352a0606f3d19dd8444c133226b Sep 16 01:20:44.514897 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:44.514859 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj" event={"ID":"21d2c26b-e753-4983-9fb9-223b8a4acf32","Type":"ContainerStarted","Data":"fb1680cea54ece60b573eb0b0b565028e52da352a0606f3d19dd8444c133226b"} Sep 16 01:20:45.097006 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:45.096964 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:45.152065 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:45.152036 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:46.525365 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:46.525319 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj" event={"ID":"21d2c26b-e753-4983-9fb9-223b8a4acf32","Type":"ContainerStarted","Data":"a2983983d9074443261f3373e981ddcdd18de325fd7509ecd61d73de29363516"} Sep 16 01:20:46.546855 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:46.546810 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzmtj" podStartSLOduration=1.910114482 podStartE2EDuration="3.546795201s" totalImagesPullingTime="1.636680719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:20:46.545363333 +0000 UTC m=+276.797785936" watchObservedRunningTime="2026-09-16 01:20:46.546795201 +0000 UTC m=+276.799217857" Sep 16 01:20:46.906171 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:46.906134 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:47.006907 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.006883 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:47.222613 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.222539 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd"] Sep 16 01:20:47.226060 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.226039 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" Sep 16 01:20:47.228289 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.228267 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 01:20:47.228402 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.228342 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hgbd4\"" Sep 16 01:20:47.228456 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.228398 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 01:20:47.235121 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.235099 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd"] Sep 16 01:20:47.269651 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.269629 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45fcfec2-43d2-42b4-a312-7d12b272faf6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd\" (UID: \"45fcfec2-43d2-42b4-a312-7d12b272faf6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" Sep 16 01:20:47.269773 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.269661 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45fcfec2-43d2-42b4-a312-7d12b272faf6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd\" (UID: \"45fcfec2-43d2-42b4-a312-7d12b272faf6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" Sep 16 01:20:47.269773 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.269700 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rntzn\" (UniqueName: \"kubernetes.io/projected/45fcfec2-43d2-42b4-a312-7d12b272faf6-kube-api-access-rntzn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd\" (UID: \"45fcfec2-43d2-42b4-a312-7d12b272faf6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" Sep 16 01:20:47.370652 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.370629 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rntzn\" (UniqueName: \"kubernetes.io/projected/45fcfec2-43d2-42b4-a312-7d12b272faf6-kube-api-access-rntzn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd\" (UID: \"45fcfec2-43d2-42b4-a312-7d12b272faf6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" Sep 16 01:20:47.370752 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.370675 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45fcfec2-43d2-42b4-a312-7d12b272faf6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd\" (UID: \"45fcfec2-43d2-42b4-a312-7d12b272faf6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" Sep 16 01:20:47.370752 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.370700 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45fcfec2-43d2-42b4-a312-7d12b272faf6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd\" (UID: \"45fcfec2-43d2-42b4-a312-7d12b272faf6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" Sep 16 01:20:47.371048 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.371029 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45fcfec2-43d2-42b4-a312-7d12b272faf6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd\" (UID: \"45fcfec2-43d2-42b4-a312-7d12b272faf6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" Sep 16 01:20:47.371122 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.371070 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45fcfec2-43d2-42b4-a312-7d12b272faf6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd\" (UID: \"45fcfec2-43d2-42b4-a312-7d12b272faf6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" Sep 16 01:20:47.386064 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.386030 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:47.387589 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.387568 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rntzn\" (UniqueName: \"kubernetes.io/projected/45fcfec2-43d2-42b4-a312-7d12b272faf6-kube-api-access-rntzn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd\" (UID: \"45fcfec2-43d2-42b4-a312-7d12b272faf6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" Sep 16 01:20:47.480056 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.479998 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:47.535490 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.535469 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" Sep 16 01:20:47.695863 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.695833 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8sqx5"] Sep 16 01:20:47.699617 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.699599 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8sqx5" Sep 16 01:20:47.701829 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.701798 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 16 01:20:47.701951 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.701922 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 16 01:20:47.702103 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.702082 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-w59sr\"" Sep 16 01:20:47.713761 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.713742 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8sqx5"] Sep 16 01:20:47.774080 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.774016 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5dvb\" (UniqueName: \"kubernetes.io/projected/cda15eb2-9149-489a-b9a1-9af0f244cb10-kube-api-access-n5dvb\") pod \"cert-manager-webhook-6486ffd997-8sqx5\" (UID: \"cda15eb2-9149-489a-b9a1-9af0f244cb10\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8sqx5" Sep 16 01:20:47.774080 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.774058 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cda15eb2-9149-489a-b9a1-9af0f244cb10-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8sqx5\" (UID: \"cda15eb2-9149-489a-b9a1-9af0f244cb10\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8sqx5" Sep 16 01:20:47.806755 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.806727 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd"] Sep 16 01:20:47.809692 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:20:47.809660 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45fcfec2_43d2_42b4_a312_7d12b272faf6.slice/crio-4fd06e533ba007a75ec5670277eb307b24d5bef99a88c6de3ed0a60acee93787 WatchSource:0}: Error finding container 4fd06e533ba007a75ec5670277eb307b24d5bef99a88c6de3ed0a60acee93787: Status 404 returned error can't find the container with id 4fd06e533ba007a75ec5670277eb307b24d5bef99a88c6de3ed0a60acee93787 Sep 16 01:20:47.875379 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.875355 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5dvb\" (UniqueName: \"kubernetes.io/projected/cda15eb2-9149-489a-b9a1-9af0f244cb10-kube-api-access-n5dvb\") pod \"cert-manager-webhook-6486ffd997-8sqx5\" (UID: \"cda15eb2-9149-489a-b9a1-9af0f244cb10\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8sqx5" Sep 16 01:20:47.875498 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.875397 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cda15eb2-9149-489a-b9a1-9af0f244cb10-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8sqx5\" (UID: \"cda15eb2-9149-489a-b9a1-9af0f244cb10\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8sqx5" Sep 16 01:20:47.883340 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.883317 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cda15eb2-9149-489a-b9a1-9af0f244cb10-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8sqx5\" (UID: \"cda15eb2-9149-489a-b9a1-9af0f244cb10\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8sqx5" Sep 16 01:20:47.883534 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:47.883517 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5dvb\" (UniqueName: \"kubernetes.io/projected/cda15eb2-9149-489a-b9a1-9af0f244cb10-kube-api-access-n5dvb\") pod \"cert-manager-webhook-6486ffd997-8sqx5\" (UID: \"cda15eb2-9149-489a-b9a1-9af0f244cb10\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8sqx5" Sep 16 01:20:48.009233 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:48.009207 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8sqx5" Sep 16 01:20:48.279480 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:48.279453 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8sqx5"] Sep 16 01:20:48.281832 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:20:48.281798 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcda15eb2_9149_489a_b9a1_9af0f244cb10.slice/crio-8710130d7cf381714f6f28e81025d16ce484a14c6c8dc2d8ccd682112a675c46 WatchSource:0}: Error finding container 8710130d7cf381714f6f28e81025d16ce484a14c6c8dc2d8ccd682112a675c46: Status 404 returned error can't find the container with id 8710130d7cf381714f6f28e81025d16ce484a14c6c8dc2d8ccd682112a675c46 Sep 16 01:20:48.540216 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:48.540176 2558 generic.go:356] "Generic (PLEG): container finished" podID="45fcfec2-43d2-42b4-a312-7d12b272faf6" containerID="507b161b28dd5fa1c93b4d3d3c53b479b511755a8ce106165004c5ffc25741dd" exitCode=0 Sep 16 01:20:48.540596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:48.540274 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" event={"ID":"45fcfec2-43d2-42b4-a312-7d12b272faf6","Type":"ContainerDied","Data":"507b161b28dd5fa1c93b4d3d3c53b479b511755a8ce106165004c5ffc25741dd"} Sep 16 01:20:48.540596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:48.540332 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" event={"ID":"45fcfec2-43d2-42b4-a312-7d12b272faf6","Type":"ContainerStarted","Data":"4fd06e533ba007a75ec5670277eb307b24d5bef99a88c6de3ed0a60acee93787"} Sep 16 01:20:48.541643 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:48.541608 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8sqx5" event={"ID":"cda15eb2-9149-489a-b9a1-9af0f244cb10","Type":"ContainerStarted","Data":"8710130d7cf381714f6f28e81025d16ce484a14c6c8dc2d8ccd682112a675c46"} Sep 16 01:20:49.798509 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:49.798475 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:50.212175 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:50.212121 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:52.096708 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:52.096639 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:52.152723 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:52.152693 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:52.562243 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:52.562192 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8sqx5" event={"ID":"cda15eb2-9149-489a-b9a1-9af0f244cb10","Type":"ContainerStarted","Data":"35b991c1ccb4c9a2712ccd53a17f0f17d49b1ebe8fae14d0eff8692107274bde"} Sep 16 01:20:52.562395 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:52.562266 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8sqx5" Sep 16 01:20:52.564041 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:52.564019 2558 generic.go:356] "Generic (PLEG): container finished" podID="45fcfec2-43d2-42b4-a312-7d12b272faf6" containerID="c3e6a5ceab90c549a462722a4fcf1ff00031f0caf92a3bcc268ce3da6fdd0c95" exitCode=0 Sep 16 01:20:52.564137 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:52.564059 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" event={"ID":"45fcfec2-43d2-42b4-a312-7d12b272faf6","Type":"ContainerDied","Data":"c3e6a5ceab90c549a462722a4fcf1ff00031f0caf92a3bcc268ce3da6fdd0c95"} Sep 16 01:20:52.577353 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:52.577309 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-8sqx5" podStartSLOduration=2.060704826 podStartE2EDuration="5.577294337s" totalImagesPullingTime="3.516589511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:20:52.577241883 +0000 UTC m=+282.829664473" watchObservedRunningTime="2026-09-16 01:20:52.577294337 +0000 UTC m=+282.829716926" Sep 16 01:20:53.571288 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:53.571249 2558 generic.go:356] "Generic (PLEG): container finished" podID="45fcfec2-43d2-42b4-a312-7d12b272faf6" containerID="2d4f0e32096bc295f0f86b64003e5ca04e045e0068e78071cbdf83883d6bf6ff" exitCode=0 Sep 16 01:20:53.571650 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:53.571333 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" event={"ID":"45fcfec2-43d2-42b4-a312-7d12b272faf6","Type":"ContainerDied","Data":"2d4f0e32096bc295f0f86b64003e5ca04e045e0068e78071cbdf83883d6bf6ff"} Sep 16 01:20:54.699642 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:54.699620 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" Sep 16 01:20:54.733942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:54.733918 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45fcfec2-43d2-42b4-a312-7d12b272faf6-bundle\") pod \"45fcfec2-43d2-42b4-a312-7d12b272faf6\" (UID: \"45fcfec2-43d2-42b4-a312-7d12b272faf6\") " Sep 16 01:20:54.734081 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:54.733957 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45fcfec2-43d2-42b4-a312-7d12b272faf6-util\") pod \"45fcfec2-43d2-42b4-a312-7d12b272faf6\" (UID: \"45fcfec2-43d2-42b4-a312-7d12b272faf6\") " Sep 16 01:20:54.734081 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:54.733990 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rntzn\" (UniqueName: \"kubernetes.io/projected/45fcfec2-43d2-42b4-a312-7d12b272faf6-kube-api-access-rntzn\") pod \"45fcfec2-43d2-42b4-a312-7d12b272faf6\" (UID: \"45fcfec2-43d2-42b4-a312-7d12b272faf6\") " Sep 16 01:20:54.734339 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:54.734315 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fcfec2-43d2-42b4-a312-7d12b272faf6-bundle" (OuterVolumeSpecName: "bundle") pod "45fcfec2-43d2-42b4-a312-7d12b272faf6" (UID: "45fcfec2-43d2-42b4-a312-7d12b272faf6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:20:54.736241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:54.736213 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fcfec2-43d2-42b4-a312-7d12b272faf6-kube-api-access-rntzn" (OuterVolumeSpecName: "kube-api-access-rntzn") pod "45fcfec2-43d2-42b4-a312-7d12b272faf6" (UID: "45fcfec2-43d2-42b4-a312-7d12b272faf6"). InnerVolumeSpecName "kube-api-access-rntzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:20:54.738517 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:54.738492 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fcfec2-43d2-42b4-a312-7d12b272faf6-util" (OuterVolumeSpecName: "util") pod "45fcfec2-43d2-42b4-a312-7d12b272faf6" (UID: "45fcfec2-43d2-42b4-a312-7d12b272faf6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:20:54.834626 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:54.834563 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45fcfec2-43d2-42b4-a312-7d12b272faf6-bundle\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:20:54.834626 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:54.834589 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45fcfec2-43d2-42b4-a312-7d12b272faf6-util\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:20:54.834626 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:54.834601 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rntzn\" (UniqueName: \"kubernetes.io/projected/45fcfec2-43d2-42b4-a312-7d12b272faf6-kube-api-access-rntzn\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:20:55.582599 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:55.582573 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" Sep 16 01:20:55.582599 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:55.582586 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9gfqd" event={"ID":"45fcfec2-43d2-42b4-a312-7d12b272faf6","Type":"ContainerDied","Data":"4fd06e533ba007a75ec5670277eb307b24d5bef99a88c6de3ed0a60acee93787"} Sep 16 01:20:55.582793 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:55.582624 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fd06e533ba007a75ec5670277eb307b24d5bef99a88c6de3ed0a60acee93787" Sep 16 01:20:57.326605 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:57.326569 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:57.424541 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:57.424507 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:57.476971 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:57.476941 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:58.573761 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:58.573731 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8sqx5" Sep 16 01:20:59.568131 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:59.568046 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:20:59.667262 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:20:59.667230 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:00.386592 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.386561 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ggkwf"] Sep 16 01:21:00.386828 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.386817 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45fcfec2-43d2-42b4-a312-7d12b272faf6" containerName="util" Sep 16 01:21:00.386872 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.386830 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45fcfec2-43d2-42b4-a312-7d12b272faf6" containerName="util" Sep 16 01:21:00.386872 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.386843 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45fcfec2-43d2-42b4-a312-7d12b272faf6" containerName="pull" Sep 16 01:21:00.386872 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.386848 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45fcfec2-43d2-42b4-a312-7d12b272faf6" containerName="pull" Sep 16 01:21:00.386872 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.386856 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45fcfec2-43d2-42b4-a312-7d12b272faf6" containerName="extract" Sep 16 01:21:00.386872 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.386861 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45fcfec2-43d2-42b4-a312-7d12b272faf6" containerName="extract" Sep 16 01:21:00.387022 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.386916 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="45fcfec2-43d2-42b4-a312-7d12b272faf6" containerName="extract" Sep 16 01:21:00.391252 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.391237 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ggkwf" Sep 16 01:21:00.393089 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.393069 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-f9pft\"" Sep 16 01:21:00.397073 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.397046 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ggkwf"] Sep 16 01:21:00.476412 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.476385 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7a9e353-2353-46b8-a7ce-07ff487732bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-ggkwf\" (UID: \"a7a9e353-2353-46b8-a7ce-07ff487732bf\") " pod="cert-manager/cert-manager-55b4d954d5-ggkwf" Sep 16 01:21:00.476545 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.476425 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcsh8\" (UniqueName: \"kubernetes.io/projected/a7a9e353-2353-46b8-a7ce-07ff487732bf-kube-api-access-wcsh8\") pod \"cert-manager-55b4d954d5-ggkwf\" (UID: \"a7a9e353-2353-46b8-a7ce-07ff487732bf\") " pod="cert-manager/cert-manager-55b4d954d5-ggkwf" Sep 16 01:21:00.576801 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.576776 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7a9e353-2353-46b8-a7ce-07ff487732bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-ggkwf\" (UID: \"a7a9e353-2353-46b8-a7ce-07ff487732bf\") " pod="cert-manager/cert-manager-55b4d954d5-ggkwf" Sep 16 01:21:00.576919 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.576815 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcsh8\" (UniqueName: \"kubernetes.io/projected/a7a9e353-2353-46b8-a7ce-07ff487732bf-kube-api-access-wcsh8\") pod \"cert-manager-55b4d954d5-ggkwf\" (UID: \"a7a9e353-2353-46b8-a7ce-07ff487732bf\") " pod="cert-manager/cert-manager-55b4d954d5-ggkwf" Sep 16 01:21:00.590007 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.589969 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcsh8\" (UniqueName: \"kubernetes.io/projected/a7a9e353-2353-46b8-a7ce-07ff487732bf-kube-api-access-wcsh8\") pod \"cert-manager-55b4d954d5-ggkwf\" (UID: \"a7a9e353-2353-46b8-a7ce-07ff487732bf\") " pod="cert-manager/cert-manager-55b4d954d5-ggkwf" Sep 16 01:21:00.590435 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.590413 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7a9e353-2353-46b8-a7ce-07ff487732bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-ggkwf\" (UID: \"a7a9e353-2353-46b8-a7ce-07ff487732bf\") " pod="cert-manager/cert-manager-55b4d954d5-ggkwf" Sep 16 01:21:00.701285 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.701198 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ggkwf" Sep 16 01:21:00.971082 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:00.971060 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ggkwf"] Sep 16 01:21:00.973168 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:21:00.973118 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7a9e353_2353_46b8_a7ce_07ff487732bf.slice/crio-0d1f861e37fb1752db7eb0124f7d0acab73e884c7be943347fefcf18007d8683 WatchSource:0}: Error finding container 0d1f861e37fb1752db7eb0124f7d0acab73e884c7be943347fefcf18007d8683: Status 404 returned error can't find the container with id 0d1f861e37fb1752db7eb0124f7d0acab73e884c7be943347fefcf18007d8683 Sep 16 01:21:01.616713 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:01.616678 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ggkwf" event={"ID":"a7a9e353-2353-46b8-a7ce-07ff487732bf","Type":"ContainerStarted","Data":"79361686a6553c0394ac2d4898a0df442c3628cba353080fa9565d7f3b9fbcc5"} Sep 16 01:21:01.616713 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:01.616720 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ggkwf" event={"ID":"a7a9e353-2353-46b8-a7ce-07ff487732bf","Type":"ContainerStarted","Data":"0d1f861e37fb1752db7eb0124f7d0acab73e884c7be943347fefcf18007d8683"} Sep 16 01:21:01.631282 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:01.631237 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ggkwf" podStartSLOduration=1.631221654 podStartE2EDuration="1.631221654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:21:01.63048445 +0000 UTC m=+291.882907062" watchObservedRunningTime="2026-09-16 01:21:01.631221654 +0000 UTC m=+291.883644243" Sep 16 01:21:06.050965 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:06.050926 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:06.156657 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:06.156627 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:07.530396 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.530370 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t"] Sep 16 01:21:07.535242 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.533700 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" Sep 16 01:21:07.538420 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.536304 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hgbd4\"" Sep 16 01:21:07.538420 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.536640 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 01:21:07.538420 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.536916 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 01:21:07.541793 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.541768 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t"] Sep 16 01:21:07.628750 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.628722 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da226be3-ac00-475e-bd9c-48bf5ae8bd46-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t\" (UID: \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" Sep 16 01:21:07.628881 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.628771 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggkv\" (UniqueName: \"kubernetes.io/projected/da226be3-ac00-475e-bd9c-48bf5ae8bd46-kube-api-access-6ggkv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t\" (UID: \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" Sep 16 01:21:07.628881 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.628844 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da226be3-ac00-475e-bd9c-48bf5ae8bd46-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t\" (UID: \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" Sep 16 01:21:07.729906 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.729871 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da226be3-ac00-475e-bd9c-48bf5ae8bd46-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t\" (UID: \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" Sep 16 01:21:07.730025 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.729923 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da226be3-ac00-475e-bd9c-48bf5ae8bd46-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t\" (UID: \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" Sep 16 01:21:07.730025 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.729960 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggkv\" (UniqueName: \"kubernetes.io/projected/da226be3-ac00-475e-bd9c-48bf5ae8bd46-kube-api-access-6ggkv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t\" (UID: \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" Sep 16 01:21:07.730264 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.730245 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da226be3-ac00-475e-bd9c-48bf5ae8bd46-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t\" (UID: \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" Sep 16 01:21:07.730341 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.730322 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da226be3-ac00-475e-bd9c-48bf5ae8bd46-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t\" (UID: \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" Sep 16 01:21:07.737424 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.737394 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggkv\" (UniqueName: \"kubernetes.io/projected/da226be3-ac00-475e-bd9c-48bf5ae8bd46-kube-api-access-6ggkv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t\" (UID: \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" Sep 16 01:21:07.846540 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:07.846484 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" Sep 16 01:21:08.115361 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:08.115333 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t"] Sep 16 01:21:08.118105 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:21:08.118076 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda226be3_ac00_475e_bd9c_48bf5ae8bd46.slice/crio-90b8d6a13b8674831659f75517e9bcf39a1cc09a3c05d2ff1c1f30cc78f75bd9 WatchSource:0}: Error finding container 90b8d6a13b8674831659f75517e9bcf39a1cc09a3c05d2ff1c1f30cc78f75bd9: Status 404 returned error can't find the container with id 90b8d6a13b8674831659f75517e9bcf39a1cc09a3c05d2ff1c1f30cc78f75bd9 Sep 16 01:21:08.283075 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:08.283046 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:08.648552 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:08.648524 2558 generic.go:356] "Generic (PLEG): container finished" podID="da226be3-ac00-475e-bd9c-48bf5ae8bd46" containerID="f7cbbcbc0c138a2f8fef22117ce6e7d8c9ce2adb3f932e93a55319701ec03c4d" exitCode=0 Sep 16 01:21:08.648907 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:08.648563 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" event={"ID":"da226be3-ac00-475e-bd9c-48bf5ae8bd46","Type":"ContainerDied","Data":"f7cbbcbc0c138a2f8fef22117ce6e7d8c9ce2adb3f932e93a55319701ec03c4d"} Sep 16 01:21:08.648907 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:08.648596 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" event={"ID":"da226be3-ac00-475e-bd9c-48bf5ae8bd46","Type":"ContainerStarted","Data":"90b8d6a13b8674831659f75517e9bcf39a1cc09a3c05d2ff1c1f30cc78f75bd9"} Sep 16 01:21:09.654596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:09.654565 2558 generic.go:356] "Generic (PLEG): container finished" podID="da226be3-ac00-475e-bd9c-48bf5ae8bd46" containerID="b8abad646298760f0aaee6b17ea04e058993430542b4354933d6926f68e64c12" exitCode=0 Sep 16 01:21:09.654949 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:09.654658 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" event={"ID":"da226be3-ac00-475e-bd9c-48bf5ae8bd46","Type":"ContainerDied","Data":"b8abad646298760f0aaee6b17ea04e058993430542b4354933d6926f68e64c12"} Sep 16 01:21:10.169039 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:10.169012 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:21:10.169235 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:10.169108 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:21:10.179589 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:10.179566 2558 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 01:21:10.660979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:10.660944 2558 generic.go:356] "Generic (PLEG): container finished" podID="da226be3-ac00-475e-bd9c-48bf5ae8bd46" containerID="a85e70d6345b0abe2867e29293831cd679eb5b7b4575cec07cb51af3407247e9" exitCode=0 Sep 16 01:21:10.662346 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:10.661033 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" event={"ID":"da226be3-ac00-475e-bd9c-48bf5ae8bd46","Type":"ContainerDied","Data":"a85e70d6345b0abe2867e29293831cd679eb5b7b4575cec07cb51af3407247e9"} Sep 16 01:21:11.787196 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:11.787174 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" Sep 16 01:21:11.861517 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:11.861487 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da226be3-ac00-475e-bd9c-48bf5ae8bd46-util\") pod \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\" (UID: \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\") " Sep 16 01:21:11.861673 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:11.861560 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da226be3-ac00-475e-bd9c-48bf5ae8bd46-bundle\") pod \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\" (UID: \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\") " Sep 16 01:21:11.861673 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:11.861612 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ggkv\" (UniqueName: \"kubernetes.io/projected/da226be3-ac00-475e-bd9c-48bf5ae8bd46-kube-api-access-6ggkv\") pod \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\" (UID: \"da226be3-ac00-475e-bd9c-48bf5ae8bd46\") " Sep 16 01:21:11.862275 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:11.862251 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da226be3-ac00-475e-bd9c-48bf5ae8bd46-bundle" (OuterVolumeSpecName: "bundle") pod "da226be3-ac00-475e-bd9c-48bf5ae8bd46" (UID: "da226be3-ac00-475e-bd9c-48bf5ae8bd46"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:21:11.863735 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:11.863709 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da226be3-ac00-475e-bd9c-48bf5ae8bd46-kube-api-access-6ggkv" (OuterVolumeSpecName: "kube-api-access-6ggkv") pod "da226be3-ac00-475e-bd9c-48bf5ae8bd46" (UID: "da226be3-ac00-475e-bd9c-48bf5ae8bd46"). InnerVolumeSpecName "kube-api-access-6ggkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:21:11.867250 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:11.867229 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da226be3-ac00-475e-bd9c-48bf5ae8bd46-util" (OuterVolumeSpecName: "util") pod "da226be3-ac00-475e-bd9c-48bf5ae8bd46" (UID: "da226be3-ac00-475e-bd9c-48bf5ae8bd46"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:21:11.962263 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:11.962211 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da226be3-ac00-475e-bd9c-48bf5ae8bd46-util\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:21:11.962263 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:11.962232 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da226be3-ac00-475e-bd9c-48bf5ae8bd46-bundle\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:21:11.962263 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:11.962242 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ggkv\" (UniqueName: \"kubernetes.io/projected/da226be3-ac00-475e-bd9c-48bf5ae8bd46-kube-api-access-6ggkv\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:21:12.672081 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:12.672052 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" Sep 16 01:21:12.672253 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:12.672052 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdpp9t" event={"ID":"da226be3-ac00-475e-bd9c-48bf5ae8bd46","Type":"ContainerDied","Data":"90b8d6a13b8674831659f75517e9bcf39a1cc09a3c05d2ff1c1f30cc78f75bd9"} Sep 16 01:21:12.672253 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:12.672198 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90b8d6a13b8674831659f75517e9bcf39a1cc09a3c05d2ff1c1f30cc78f75bd9" Sep 16 01:21:13.402890 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:13.402839 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:23.725019 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:23.724987 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:23.830451 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:23.830424 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:24.340708 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.340677 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:24.595877 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.595796 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:24.645247 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.645207 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m"] Sep 16 01:21:24.645608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.645578 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da226be3-ac00-475e-bd9c-48bf5ae8bd46" containerName="extract" Sep 16 01:21:24.645608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.645601 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="da226be3-ac00-475e-bd9c-48bf5ae8bd46" containerName="extract" Sep 16 01:21:24.645608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.645613 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da226be3-ac00-475e-bd9c-48bf5ae8bd46" containerName="pull" Sep 16 01:21:24.645815 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.645621 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="da226be3-ac00-475e-bd9c-48bf5ae8bd46" containerName="pull" Sep 16 01:21:24.645815 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.645644 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da226be3-ac00-475e-bd9c-48bf5ae8bd46" containerName="util" Sep 16 01:21:24.645815 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.645651 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="da226be3-ac00-475e-bd9c-48bf5ae8bd46" containerName="util" Sep 16 01:21:24.645815 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.645706 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="da226be3-ac00-475e-bd9c-48bf5ae8bd46" containerName="extract" Sep 16 01:21:24.650318 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.650296 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.653224 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.653203 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-v9c2d\"" Sep 16 01:21:24.653321 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.653203 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 16 01:21:24.653321 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.653259 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 16 01:21:24.653321 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.653203 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 16 01:21:24.653321 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.653205 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 16 01:21:24.653527 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.653334 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 16 01:21:24.661051 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.661031 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m"] Sep 16 01:21:24.698075 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.698048 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:24.750107 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.750071 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwx6\" (UniqueName: \"kubernetes.io/projected/29412d1b-b3cb-45e9-b326-53469931bf2d-kube-api-access-mdwx6\") pod \"lws-controller-manager-65dff4dd97-b967m\" (UID: \"29412d1b-b3cb-45e9-b326-53469931bf2d\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.750107 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.750106 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/29412d1b-b3cb-45e9-b326-53469931bf2d-metrics-cert\") pod \"lws-controller-manager-65dff4dd97-b967m\" (UID: \"29412d1b-b3cb-45e9-b326-53469931bf2d\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.750505 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.750127 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29412d1b-b3cb-45e9-b326-53469931bf2d-cert\") pod \"lws-controller-manager-65dff4dd97-b967m\" (UID: \"29412d1b-b3cb-45e9-b326-53469931bf2d\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.750505 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.750149 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/29412d1b-b3cb-45e9-b326-53469931bf2d-manager-config\") pod \"lws-controller-manager-65dff4dd97-b967m\" (UID: \"29412d1b-b3cb-45e9-b326-53469931bf2d\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.810313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.810283 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp"] Sep 16 01:21:24.812714 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.812695 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:24.819973 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.819953 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 16 01:21:24.820149 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.820130 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 16 01:21:24.820229 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.820201 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 16 01:21:24.826917 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.826901 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 16 01:21:24.830374 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.830360 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8s6lc\"" Sep 16 01:21:24.850778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.850720 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29412d1b-b3cb-45e9-b326-53469931bf2d-cert\") pod \"lws-controller-manager-65dff4dd97-b967m\" (UID: \"29412d1b-b3cb-45e9-b326-53469931bf2d\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.850778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.850749 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/29412d1b-b3cb-45e9-b326-53469931bf2d-manager-config\") pod \"lws-controller-manager-65dff4dd97-b967m\" (UID: \"29412d1b-b3cb-45e9-b326-53469931bf2d\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.850923 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.850834 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwx6\" (UniqueName: \"kubernetes.io/projected/29412d1b-b3cb-45e9-b326-53469931bf2d-kube-api-access-mdwx6\") pod \"lws-controller-manager-65dff4dd97-b967m\" (UID: \"29412d1b-b3cb-45e9-b326-53469931bf2d\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.850923 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.850861 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/29412d1b-b3cb-45e9-b326-53469931bf2d-metrics-cert\") pod \"lws-controller-manager-65dff4dd97-b967m\" (UID: \"29412d1b-b3cb-45e9-b326-53469931bf2d\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.851657 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.851636 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/29412d1b-b3cb-45e9-b326-53469931bf2d-manager-config\") pod \"lws-controller-manager-65dff4dd97-b967m\" (UID: \"29412d1b-b3cb-45e9-b326-53469931bf2d\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.853901 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.853879 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29412d1b-b3cb-45e9-b326-53469931bf2d-cert\") pod \"lws-controller-manager-65dff4dd97-b967m\" (UID: \"29412d1b-b3cb-45e9-b326-53469931bf2d\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.854010 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.853963 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/29412d1b-b3cb-45e9-b326-53469931bf2d-metrics-cert\") pod \"lws-controller-manager-65dff4dd97-b967m\" (UID: \"29412d1b-b3cb-45e9-b326-53469931bf2d\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.854436 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.854414 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp"] Sep 16 01:21:24.877984 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.877956 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwx6\" (UniqueName: \"kubernetes.io/projected/29412d1b-b3cb-45e9-b326-53469931bf2d-kube-api-access-mdwx6\") pod \"lws-controller-manager-65dff4dd97-b967m\" (UID: \"29412d1b-b3cb-45e9-b326-53469931bf2d\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.917939 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.917911 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt"] Sep 16 01:21:24.920361 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.920345 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" Sep 16 01:21:24.922669 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.922650 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 01:21:24.923345 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.923328 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hgbd4\"" Sep 16 01:21:24.923427 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.923398 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 01:21:24.933377 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.933357 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt"] Sep 16 01:21:24.951362 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.951338 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zkz7\" (UniqueName: \"kubernetes.io/projected/01611d7f-9833-4b82-986d-61ce785f0a94-kube-api-access-7zkz7\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:24.951465 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.951373 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01611d7f-9833-4b82-986d-61ce785f0a94-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:24.951465 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.951394 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01611d7f-9833-4b82-986d-61ce785f0a94-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:24.951465 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.951448 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01611d7f-9833-4b82-986d-61ce785f0a94-manifests\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:24.951578 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.951475 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01611d7f-9833-4b82-986d-61ce785f0a94-tmp\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:24.960963 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.960931 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:24.981179 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:24.981112 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:25.052400 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.052373 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdb682cb-a7f1-4f71-972f-3144d1f20f68-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt\" (UID: \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" Sep 16 01:21:25.052545 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.052406 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdb682cb-a7f1-4f71-972f-3144d1f20f68-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt\" (UID: \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" Sep 16 01:21:25.052545 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.052433 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zkz7\" (UniqueName: \"kubernetes.io/projected/01611d7f-9833-4b82-986d-61ce785f0a94-kube-api-access-7zkz7\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:25.052545 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.052467 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01611d7f-9833-4b82-986d-61ce785f0a94-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:25.052545 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.052535 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01611d7f-9833-4b82-986d-61ce785f0a94-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:25.052727 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.052671 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01611d7f-9833-4b82-986d-61ce785f0a94-manifests\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:25.052727 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.052713 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn4sd\" (UniqueName: \"kubernetes.io/projected/cdb682cb-a7f1-4f71-972f-3144d1f20f68-kube-api-access-jn4sd\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt\" (UID: \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" Sep 16 01:21:25.052823 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.052742 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01611d7f-9833-4b82-986d-61ce785f0a94-tmp\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:25.053029 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.052994 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01611d7f-9833-4b82-986d-61ce785f0a94-manifests\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:25.054886 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.054864 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01611d7f-9833-4b82-986d-61ce785f0a94-tmp\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:25.055048 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.055032 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01611d7f-9833-4b82-986d-61ce785f0a94-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:25.055237 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.055220 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01611d7f-9833-4b82-986d-61ce785f0a94-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:25.060084 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.060062 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zkz7\" (UniqueName: \"kubernetes.io/projected/01611d7f-9833-4b82-986d-61ce785f0a94-kube-api-access-7zkz7\") pod \"opendatahub-operator-controller-manager-5f84c66984-k7wxp\" (UID: \"01611d7f-9833-4b82-986d-61ce785f0a94\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:25.076079 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.076052 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:25.123968 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.123947 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:25.153981 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.153944 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdb682cb-a7f1-4f71-972f-3144d1f20f68-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt\" (UID: \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" Sep 16 01:21:25.154136 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.153995 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdb682cb-a7f1-4f71-972f-3144d1f20f68-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt\" (UID: \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" Sep 16 01:21:25.154136 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.154077 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4sd\" (UniqueName: \"kubernetes.io/projected/cdb682cb-a7f1-4f71-972f-3144d1f20f68-kube-api-access-jn4sd\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt\" (UID: \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" Sep 16 01:21:25.154422 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.154391 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdb682cb-a7f1-4f71-972f-3144d1f20f68-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt\" (UID: \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" Sep 16 01:21:25.154487 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.154414 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdb682cb-a7f1-4f71-972f-3144d1f20f68-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt\" (UID: \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" Sep 16 01:21:25.162707 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.162684 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn4sd\" (UniqueName: \"kubernetes.io/projected/cdb682cb-a7f1-4f71-972f-3144d1f20f68-kube-api-access-jn4sd\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt\" (UID: \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" Sep 16 01:21:25.177080 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.177051 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:25.229715 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.229688 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" Sep 16 01:21:25.252442 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.252420 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m"] Sep 16 01:21:25.254751 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:21:25.254721 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29412d1b_b3cb_45e9_b326_53469931bf2d.slice/crio-32aaace7a442240e693b22366fded59e0ecfb089755da7f6439880dab80b3d45 WatchSource:0}: Error finding container 32aaace7a442240e693b22366fded59e0ecfb089755da7f6439880dab80b3d45: Status 404 returned error can't find the container with id 32aaace7a442240e693b22366fded59e0ecfb089755da7f6439880dab80b3d45 Sep 16 01:21:25.256746 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.256728 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 01:21:25.288816 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.288791 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:25.388899 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.388831 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:25.402184 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.402138 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp"] Sep 16 01:21:25.405106 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:21:25.405086 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01611d7f_9833_4b82_986d_61ce785f0a94.slice/crio-989cc238334357179c8fb8df0ea46b07947643c553c64fddf0ecd5e778d42e4a WatchSource:0}: Error finding container 989cc238334357179c8fb8df0ea46b07947643c553c64fddf0ecd5e778d42e4a: Status 404 returned error can't find the container with id 989cc238334357179c8fb8df0ea46b07947643c553c64fddf0ecd5e778d42e4a Sep 16 01:21:25.487913 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.487882 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:25.501856 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.501833 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt"] Sep 16 01:21:25.504912 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:21:25.504886 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb682cb_a7f1_4f71_972f_3144d1f20f68.slice/crio-1046dd69709f3bed673eed3a192f2b7a4123851af3232440d57c6d7177cbcd2a WatchSource:0}: Error finding container 1046dd69709f3bed673eed3a192f2b7a4123851af3232440d57c6d7177cbcd2a: Status 404 returned error can't find the container with id 1046dd69709f3bed673eed3a192f2b7a4123851af3232440d57c6d7177cbcd2a Sep 16 01:21:25.729060 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.728972 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" event={"ID":"01611d7f-9833-4b82-986d-61ce785f0a94","Type":"ContainerStarted","Data":"989cc238334357179c8fb8df0ea46b07947643c553c64fddf0ecd5e778d42e4a"} Sep 16 01:21:25.730474 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.730452 2558 generic.go:356] "Generic (PLEG): container finished" podID="cdb682cb-a7f1-4f71-972f-3144d1f20f68" containerID="0fbd09388f156330e1a28970553b059ac45b28f144c4f9bb57041732c654768d" exitCode=0 Sep 16 01:21:25.730591 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.730546 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" event={"ID":"cdb682cb-a7f1-4f71-972f-3144d1f20f68","Type":"ContainerDied","Data":"0fbd09388f156330e1a28970553b059ac45b28f144c4f9bb57041732c654768d"} Sep 16 01:21:25.730591 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.730587 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" event={"ID":"cdb682cb-a7f1-4f71-972f-3144d1f20f68","Type":"ContainerStarted","Data":"1046dd69709f3bed673eed3a192f2b7a4123851af3232440d57c6d7177cbcd2a"} Sep 16 01:21:25.731789 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:25.731771 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" event={"ID":"29412d1b-b3cb-45e9-b326-53469931bf2d","Type":"ContainerStarted","Data":"32aaace7a442240e693b22366fded59e0ecfb089755da7f6439880dab80b3d45"} Sep 16 01:21:26.742461 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:26.741756 2558 generic.go:356] "Generic (PLEG): container finished" podID="cdb682cb-a7f1-4f71-972f-3144d1f20f68" containerID="430591f8bfe3cef3d009e7de98ef4d09622981c9dcd614dcd99b0241438f444a" exitCode=0 Sep 16 01:21:26.742461 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:26.741822 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" event={"ID":"cdb682cb-a7f1-4f71-972f-3144d1f20f68","Type":"ContainerDied","Data":"430591f8bfe3cef3d009e7de98ef4d09622981c9dcd614dcd99b0241438f444a"} Sep 16 01:21:27.750736 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:27.750701 2558 generic.go:356] "Generic (PLEG): container finished" podID="cdb682cb-a7f1-4f71-972f-3144d1f20f68" containerID="d1c4c6b7b7f985c38bd49f90c93a52bb57e8705996ae1957c1c6cca09e8e29e9" exitCode=0 Sep 16 01:21:27.751132 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:27.750758 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" event={"ID":"cdb682cb-a7f1-4f71-972f-3144d1f20f68","Type":"ContainerDied","Data":"d1c4c6b7b7f985c38bd49f90c93a52bb57e8705996ae1957c1c6cca09e8e29e9"} Sep 16 01:21:28.891729 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:28.891705 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" Sep 16 01:21:28.986238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:28.986209 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdb682cb-a7f1-4f71-972f-3144d1f20f68-bundle\") pod \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\" (UID: \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\") " Sep 16 01:21:28.986378 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:28.986284 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn4sd\" (UniqueName: \"kubernetes.io/projected/cdb682cb-a7f1-4f71-972f-3144d1f20f68-kube-api-access-jn4sd\") pod \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\" (UID: \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\") " Sep 16 01:21:28.986378 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:28.986334 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdb682cb-a7f1-4f71-972f-3144d1f20f68-util\") pod \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\" (UID: \"cdb682cb-a7f1-4f71-972f-3144d1f20f68\") " Sep 16 01:21:29.059918 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:29.059843 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb682cb-a7f1-4f71-972f-3144d1f20f68-bundle" (OuterVolumeSpecName: "bundle") pod "cdb682cb-a7f1-4f71-972f-3144d1f20f68" (UID: "cdb682cb-a7f1-4f71-972f-3144d1f20f68"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:21:29.061305 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:29.061273 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb682cb-a7f1-4f71-972f-3144d1f20f68-kube-api-access-jn4sd" (OuterVolumeSpecName: "kube-api-access-jn4sd") pod "cdb682cb-a7f1-4f71-972f-3144d1f20f68" (UID: "cdb682cb-a7f1-4f71-972f-3144d1f20f68"). InnerVolumeSpecName "kube-api-access-jn4sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:21:29.067378 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:29.067332 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb682cb-a7f1-4f71-972f-3144d1f20f68-util" (OuterVolumeSpecName: "util") pod "cdb682cb-a7f1-4f71-972f-3144d1f20f68" (UID: "cdb682cb-a7f1-4f71-972f-3144d1f20f68"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:21:29.087626 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:29.087602 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdb682cb-a7f1-4f71-972f-3144d1f20f68-bundle\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:21:29.087626 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:29.087623 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn4sd\" (UniqueName: \"kubernetes.io/projected/cdb682cb-a7f1-4f71-972f-3144d1f20f68-kube-api-access-jn4sd\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:21:29.087735 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:29.087633 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdb682cb-a7f1-4f71-972f-3144d1f20f68-util\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:21:29.764914 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:29.764865 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" event={"ID":"29412d1b-b3cb-45e9-b326-53469931bf2d","Type":"ContainerStarted","Data":"3cdea82670f266a272677457d47261eb0c8d8689f5bcc6cc1a74d0d5406b9d9c"} Sep 16 01:21:29.765104 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:29.765029 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:29.766517 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:29.766493 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" event={"ID":"01611d7f-9833-4b82-986d-61ce785f0a94","Type":"ContainerStarted","Data":"2f6cf0b5d55ac14bed374fab160d305331041e448a1d7eda6c4fa37d31e1a656"} Sep 16 01:21:29.768490 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:29.768471 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" event={"ID":"cdb682cb-a7f1-4f71-972f-3144d1f20f68","Type":"ContainerDied","Data":"1046dd69709f3bed673eed3a192f2b7a4123851af3232440d57c6d7177cbcd2a"} Sep 16 01:21:29.768579 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:29.768500 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b098xnt" Sep 16 01:21:29.768619 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:29.768506 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1046dd69709f3bed673eed3a192f2b7a4123851af3232440d57c6d7177cbcd2a" Sep 16 01:21:29.780695 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:29.780657 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" podStartSLOduration=2.169605234 podStartE2EDuration="5.780642486s" totalImagesPullingTime="3.611037252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:21:29.779307984 +0000 UTC m=+320.031730570" watchObservedRunningTime="2026-09-16 01:21:29.780642486 +0000 UTC m=+320.033065074" Sep 16 01:21:30.775021 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:30.774984 2558 generic.go:356] "Generic (PLEG): container finished" podID="01611d7f-9833-4b82-986d-61ce785f0a94" containerID="2f6cf0b5d55ac14bed374fab160d305331041e448a1d7eda6c4fa37d31e1a656" exitCode=0 Sep 16 01:21:30.775468 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:30.775072 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" event={"ID":"01611d7f-9833-4b82-986d-61ce785f0a94","Type":"ContainerDied","Data":"2f6cf0b5d55ac14bed374fab160d305331041e448a1d7eda6c4fa37d31e1a656"} Sep 16 01:21:31.781317 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:31.781283 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" event={"ID":"01611d7f-9833-4b82-986d-61ce785f0a94","Type":"ContainerStarted","Data":"b32c5c42284dd35ef69fede25c64c6028d830c6bab150615fa323dd59ef6a611"} Sep 16 01:21:31.781725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:31.781524 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:31.801239 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:31.801193 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" podStartSLOduration=2.905764123 podStartE2EDuration="7.80117906s" totalImagesPullingTime="3.876895923s" totalInitContainerRuntime="1.018519014s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:21:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:21:31.799485621 +0000 UTC m=+322.051908237" watchObservedRunningTime="2026-09-16 01:21:31.80117906 +0000 UTC m=+322.053601642" Sep 16 01:21:40.777884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:40.777851 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-b967m" Sep 16 01:21:42.789855 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:42.789825 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-k7wxp" Sep 16 01:21:44.375503 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:44.375468 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:44.453691 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:44.453665 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:47.314908 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:47.314881 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:47.529086 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:47.529051 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:48.493926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:48.493868 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:48.591457 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:48.591408 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:49.334406 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:49.334363 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:53.463768 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.463731 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:53.522484 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.522456 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:53.720938 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.720865 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw"] Sep 16 01:21:53.721254 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.721232 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb682cb-a7f1-4f71-972f-3144d1f20f68" containerName="pull" Sep 16 01:21:53.721254 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.721247 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb682cb-a7f1-4f71-972f-3144d1f20f68" containerName="pull" Sep 16 01:21:53.721254 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.721255 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb682cb-a7f1-4f71-972f-3144d1f20f68" containerName="util" Sep 16 01:21:53.721370 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.721260 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb682cb-a7f1-4f71-972f-3144d1f20f68" containerName="util" Sep 16 01:21:53.721370 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.721277 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb682cb-a7f1-4f71-972f-3144d1f20f68" containerName="extract" Sep 16 01:21:53.721370 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.721283 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb682cb-a7f1-4f71-972f-3144d1f20f68" containerName="extract" Sep 16 01:21:53.721370 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.721351 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb682cb-a7f1-4f71-972f-3144d1f20f68" containerName="extract" Sep 16 01:21:53.729205 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.729185 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.731743 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.731709 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 01:21:53.731897 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.731817 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-nn57q\"" Sep 16 01:21:53.731897 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.731873 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 01:21:53.732025 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.732007 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 01:21:53.734216 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.734195 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw"] Sep 16 01:21:53.751003 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.750974 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:53.878757 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.878729 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.878892 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.878777 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.878892 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.878824 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.878892 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.878855 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.878892 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.878887 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.879066 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.878904 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.879066 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.878956 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.879066 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.878972 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.879066 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.878987 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65qhb\" (UniqueName: \"kubernetes.io/projected/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-kube-api-access-65qhb\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.895344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.895316 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:53.980084 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980020 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.980084 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980063 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.980285 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980085 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.980285 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980104 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.980285 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980133 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.980285 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980176 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.980285 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980248 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.980285 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980273 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.980547 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980293 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65qhb\" (UniqueName: \"kubernetes.io/projected/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-kube-api-access-65qhb\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.980604 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980549 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.980604 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980588 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.980695 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980680 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.980734 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980675 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.980988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.980967 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.982528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.982511 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.982832 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.982814 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.987458 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.987437 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:53.987997 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:53.987981 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65qhb\" (UniqueName: \"kubernetes.io/projected/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-kube-api-access-65qhb\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:54.037996 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:54.037968 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:54.041804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:54.041789 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:54.315358 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:54.315302 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw"] Sep 16 01:21:54.317963 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:21:54.317931 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b21ef6_f4c1_48d4_a0ab_f524b55ca745.slice/crio-12891f8c5b6171a8f527773621f3901b8e8608518936e318e7c6edafc4b5cdac WatchSource:0}: Error finding container 12891f8c5b6171a8f527773621f3901b8e8608518936e318e7c6edafc4b5cdac: Status 404 returned error can't find the container with id 12891f8c5b6171a8f527773621f3901b8e8608518936e318e7c6edafc4b5cdac Sep 16 01:21:54.886834 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:54.886800 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" event={"ID":"12b21ef6-f4c1-48d4-a0ab-f524b55ca745","Type":"ContainerStarted","Data":"12891f8c5b6171a8f527773621f3901b8e8608518936e318e7c6edafc4b5cdac"} Sep 16 01:21:55.425309 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.425270 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lkdhm"] Sep 16 01:21:55.431321 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.431297 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" Sep 16 01:21:55.433376 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.433349 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 16 01:21:55.433523 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.433349 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-c25fb\"" Sep 16 01:21:55.437177 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.437135 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lkdhm"] Sep 16 01:21:55.508476 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.508446 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:55.560569 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.560537 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:55.595986 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.595937 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vdkz\" (UniqueName: \"kubernetes.io/projected/e0926020-5dbf-41d5-b6e0-21a08053d587-kube-api-access-2vdkz\") pod \"odh-model-controller-7998c67dc4-lkdhm\" (UID: \"e0926020-5dbf-41d5-b6e0-21a08053d587\") " pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" Sep 16 01:21:55.596147 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.596048 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0926020-5dbf-41d5-b6e0-21a08053d587-cert\") pod \"odh-model-controller-7998c67dc4-lkdhm\" (UID: \"e0926020-5dbf-41d5-b6e0-21a08053d587\") " pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" Sep 16 01:21:55.636424 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.636399 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:55.696754 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.696676 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0926020-5dbf-41d5-b6e0-21a08053d587-cert\") pod \"odh-model-controller-7998c67dc4-lkdhm\" (UID: \"e0926020-5dbf-41d5-b6e0-21a08053d587\") " pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" Sep 16 01:21:55.696898 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.696772 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vdkz\" (UniqueName: \"kubernetes.io/projected/e0926020-5dbf-41d5-b6e0-21a08053d587-kube-api-access-2vdkz\") pod \"odh-model-controller-7998c67dc4-lkdhm\" (UID: \"e0926020-5dbf-41d5-b6e0-21a08053d587\") " pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" Sep 16 01:21:55.696898 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:21:55.696849 2558 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 16 01:21:55.697011 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:21:55.696933 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0926020-5dbf-41d5-b6e0-21a08053d587-cert podName:e0926020-5dbf-41d5-b6e0-21a08053d587 nodeName:}" failed. No retries permitted until 2026-09-16 01:21:56.196911774 +0000 UTC m=+346.449334356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e0926020-5dbf-41d5-b6e0-21a08053d587-cert") pod "odh-model-controller-7998c67dc4-lkdhm" (UID: "e0926020-5dbf-41d5-b6e0-21a08053d587") : secret "odh-model-controller-webhook-cert" not found Sep 16 01:21:55.704704 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.704678 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vdkz\" (UniqueName: \"kubernetes.io/projected/e0926020-5dbf-41d5-b6e0-21a08053d587-kube-api-access-2vdkz\") pod \"odh-model-controller-7998c67dc4-lkdhm\" (UID: \"e0926020-5dbf-41d5-b6e0-21a08053d587\") " pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" Sep 16 01:21:55.868918 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:55.868885 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:56.045545 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.045462 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:56.101027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.100992 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mpg47"] Sep 16 01:21:56.104667 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.104648 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mpg47" Sep 16 01:21:56.106564 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.106541 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 16 01:21:56.106690 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.106541 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-pw98v\"" Sep 16 01:21:56.114173 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.114133 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mpg47"] Sep 16 01:21:56.201328 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.201300 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0926020-5dbf-41d5-b6e0-21a08053d587-cert\") pod \"odh-model-controller-7998c67dc4-lkdhm\" (UID: \"e0926020-5dbf-41d5-b6e0-21a08053d587\") " pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" Sep 16 01:21:56.204298 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.204273 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0926020-5dbf-41d5-b6e0-21a08053d587-cert\") pod \"odh-model-controller-7998c67dc4-lkdhm\" (UID: \"e0926020-5dbf-41d5-b6e0-21a08053d587\") " pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" Sep 16 01:21:56.290901 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.290859 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:21:56.302610 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.302535 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5252e61c-a46e-42c4-a1b5-d2f2fd9528ac-tls-certs\") pod \"model-serving-api-54d89bd5fc-mpg47\" (UID: \"5252e61c-a46e-42c4-a1b5-d2f2fd9528ac\") " pod="opendatahub/model-serving-api-54d89bd5fc-mpg47" Sep 16 01:21:56.302746 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.302612 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmzhg\" (UniqueName: \"kubernetes.io/projected/5252e61c-a46e-42c4-a1b5-d2f2fd9528ac-kube-api-access-dmzhg\") pod \"model-serving-api-54d89bd5fc-mpg47\" (UID: \"5252e61c-a46e-42c4-a1b5-d2f2fd9528ac\") " pod="opendatahub/model-serving-api-54d89bd5fc-mpg47" Sep 16 01:21:56.345839 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.345811 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" Sep 16 01:21:56.403877 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.403839 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5252e61c-a46e-42c4-a1b5-d2f2fd9528ac-tls-certs\") pod \"model-serving-api-54d89bd5fc-mpg47\" (UID: \"5252e61c-a46e-42c4-a1b5-d2f2fd9528ac\") " pod="opendatahub/model-serving-api-54d89bd5fc-mpg47" Sep 16 01:21:56.404042 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.403950 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmzhg\" (UniqueName: \"kubernetes.io/projected/5252e61c-a46e-42c4-a1b5-d2f2fd9528ac-kube-api-access-dmzhg\") pod \"model-serving-api-54d89bd5fc-mpg47\" (UID: \"5252e61c-a46e-42c4-a1b5-d2f2fd9528ac\") " pod="opendatahub/model-serving-api-54d89bd5fc-mpg47" Sep 16 01:21:56.406597 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.406574 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5252e61c-a46e-42c4-a1b5-d2f2fd9528ac-tls-certs\") pod \"model-serving-api-54d89bd5fc-mpg47\" (UID: \"5252e61c-a46e-42c4-a1b5-d2f2fd9528ac\") " pod="opendatahub/model-serving-api-54d89bd5fc-mpg47" Sep 16 01:21:56.411388 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.411356 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmzhg\" (UniqueName: \"kubernetes.io/projected/5252e61c-a46e-42c4-a1b5-d2f2fd9528ac-kube-api-access-dmzhg\") pod \"model-serving-api-54d89bd5fc-mpg47\" (UID: \"5252e61c-a46e-42c4-a1b5-d2f2fd9528ac\") " pod="opendatahub/model-serving-api-54d89bd5fc-mpg47" Sep 16 01:21:56.418063 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:56.418016 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mpg47" Sep 16 01:21:57.332241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:57.332187 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:21:57.332671 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:57.332284 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:21:57.332671 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:57.332333 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:21:57.523880 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:57.523740 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lkdhm"] Sep 16 01:21:57.526601 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:57.526582 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mpg47"] Sep 16 01:21:57.527518 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:21:57.527491 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0926020_5dbf_41d5_b6e0_21a08053d587.slice/crio-5fe00ec86ffaffa40f467cb521d7e1870f7795cd5c3dce00aa72d8f86a43fb6e WatchSource:0}: Error finding container 5fe00ec86ffaffa40f467cb521d7e1870f7795cd5c3dce00aa72d8f86a43fb6e: Status 404 returned error can't find the container with id 5fe00ec86ffaffa40f467cb521d7e1870f7795cd5c3dce00aa72d8f86a43fb6e Sep 16 01:21:57.529519 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:21:57.529494 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5252e61c_a46e_42c4_a1b5_d2f2fd9528ac.slice/crio-88bfdc9c8733eb38a2366898585978465f9ce5e07ff5dc50751aaa459bd1b3f4 WatchSource:0}: Error finding container 88bfdc9c8733eb38a2366898585978465f9ce5e07ff5dc50751aaa459bd1b3f4: Status 404 returned error can't find the container with id 88bfdc9c8733eb38a2366898585978465f9ce5e07ff5dc50751aaa459bd1b3f4 Sep 16 01:21:57.905242 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:57.905202 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" event={"ID":"12b21ef6-f4c1-48d4-a0ab-f524b55ca745","Type":"ContainerStarted","Data":"3c2405cee7882fa7852a2f28065498290b6b476a6adf764b7cc1edff0148504a"} Sep 16 01:21:57.906493 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:57.906465 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" event={"ID":"e0926020-5dbf-41d5-b6e0-21a08053d587","Type":"ContainerStarted","Data":"5fe00ec86ffaffa40f467cb521d7e1870f7795cd5c3dce00aa72d8f86a43fb6e"} Sep 16 01:21:57.907835 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:57.907797 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mpg47" event={"ID":"5252e61c-a46e-42c4-a1b5-d2f2fd9528ac","Type":"ContainerStarted","Data":"88bfdc9c8733eb38a2366898585978465f9ce5e07ff5dc50751aaa459bd1b3f4"} Sep 16 01:21:57.925338 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:57.925288 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" podStartSLOduration=1.913387144 podStartE2EDuration="4.925269018s" totalImagesPullingTime="3.011881874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:21:57.922473355 +0000 UTC m=+348.174895945" watchObservedRunningTime="2026-09-16 01:21:57.925269018 +0000 UTC m=+348.177691611" Sep 16 01:21:58.042966 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:58.042937 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:21:58.044284 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:58.044260 2558 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 16 01:21:58.044346 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:58.044316 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" podUID="12b21ef6-f4c1-48d4-a0ab-f524b55ca745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 16 01:21:58.694854 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:58.694818 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw"] Sep 16 01:21:59.043021 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:59.042939 2558 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 16 01:21:59.043210 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:59.043015 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" podUID="12b21ef6-f4c1-48d4-a0ab-f524b55ca745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 16 01:21:59.919913 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:21:59.919848 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" podUID="12b21ef6-f4c1-48d4-a0ab-f524b55ca745" containerName="istio-proxy" containerID="cri-o://3c2405cee7882fa7852a2f28065498290b6b476a6adf764b7cc1edff0148504a" gracePeriod=30 Sep 16 01:22:01.932283 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:01.932248 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" event={"ID":"e0926020-5dbf-41d5-b6e0-21a08053d587","Type":"ContainerStarted","Data":"5f5007c2395c477278792466043d0a4dedb0e8c1ea3aa0f79fb2dd0d69170f35"} Sep 16 01:22:01.932791 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:01.932359 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" Sep 16 01:22:01.933874 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:01.933847 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mpg47" event={"ID":"5252e61c-a46e-42c4-a1b5-d2f2fd9528ac","Type":"ContainerStarted","Data":"f8fb8eb92e7ccb0f1ea13379bb5d0f899e014f13efffb3484f943c9f1f9744d1"} Sep 16 01:22:01.933969 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:01.933917 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mpg47" Sep 16 01:22:01.949856 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:01.949816 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" podStartSLOduration=2.927312757 podStartE2EDuration="6.949803157s" totalImagesPullingTime="4.0224904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:22:01.947565954 +0000 UTC m=+352.199988545" watchObservedRunningTime="2026-09-16 01:22:01.949803157 +0000 UTC m=+352.202225744" Sep 16 01:22:01.964041 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:01.964000 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mpg47" podStartSLOduration=1.980310091 podStartE2EDuration="5.963987556s" totalImagesPullingTime="3.983677465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:22:01.963378214 +0000 UTC m=+352.215800809" watchObservedRunningTime="2026-09-16 01:22:01.963987556 +0000 UTC m=+352.216410167" Sep 16 01:22:04.611646 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.611615 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:04.688490 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.688461 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q42rb"] Sep 16 01:22:04.692088 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.692066 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" Sep 16 01:22:04.693238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.693213 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:04.694394 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.694372 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 16 01:22:04.694622 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.694600 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-f7vmb\"" Sep 16 01:22:04.696023 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.696003 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk"] Sep 16 01:22:04.699263 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.699245 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" Sep 16 01:22:04.701570 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.701553 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-g8mr8\"" Sep 16 01:22:04.703706 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.703684 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 16 01:22:04.705605 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.705580 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q42rb"] Sep 16 01:22:04.717369 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.717347 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk"] Sep 16 01:22:04.784846 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.784811 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5kk9\" (UniqueName: \"kubernetes.io/projected/8b44e0e4-9e34-4811-a629-0516078a58a1-kube-api-access-w5kk9\") pod \"llmisvc-controller-manager-777c6bd7d5-skkpk\" (UID: \"8b44e0e4-9e34-4811-a629-0516078a58a1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" Sep 16 01:22:04.785002 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.784869 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns7ft\" (UniqueName: \"kubernetes.io/projected/63ff281e-5d24-44a2-970c-a6ef19b8f64b-kube-api-access-ns7ft\") pod \"kserve-controller-manager-d6f4b65b8-q42rb\" (UID: \"63ff281e-5d24-44a2-970c-a6ef19b8f64b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" Sep 16 01:22:04.785002 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.784898 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b44e0e4-9e34-4811-a629-0516078a58a1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-skkpk\" (UID: \"8b44e0e4-9e34-4811-a629-0516078a58a1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" Sep 16 01:22:04.785002 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.784938 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63ff281e-5d24-44a2-970c-a6ef19b8f64b-cert\") pod \"kserve-controller-manager-d6f4b65b8-q42rb\" (UID: \"63ff281e-5d24-44a2-970c-a6ef19b8f64b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" Sep 16 01:22:04.885800 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.885766 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns7ft\" (UniqueName: \"kubernetes.io/projected/63ff281e-5d24-44a2-970c-a6ef19b8f64b-kube-api-access-ns7ft\") pod \"kserve-controller-manager-d6f4b65b8-q42rb\" (UID: \"63ff281e-5d24-44a2-970c-a6ef19b8f64b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" Sep 16 01:22:04.885937 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.885807 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b44e0e4-9e34-4811-a629-0516078a58a1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-skkpk\" (UID: \"8b44e0e4-9e34-4811-a629-0516078a58a1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" Sep 16 01:22:04.885937 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.885834 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63ff281e-5d24-44a2-970c-a6ef19b8f64b-cert\") pod \"kserve-controller-manager-d6f4b65b8-q42rb\" (UID: \"63ff281e-5d24-44a2-970c-a6ef19b8f64b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" Sep 16 01:22:04.885937 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.885918 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5kk9\" (UniqueName: \"kubernetes.io/projected/8b44e0e4-9e34-4811-a629-0516078a58a1-kube-api-access-w5kk9\") pod \"llmisvc-controller-manager-777c6bd7d5-skkpk\" (UID: \"8b44e0e4-9e34-4811-a629-0516078a58a1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" Sep 16 01:22:04.886218 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:22:04.885935 2558 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 16 01:22:04.886218 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:22:04.885983 2558 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 16 01:22:04.886218 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:22:04.886007 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b44e0e4-9e34-4811-a629-0516078a58a1-cert podName:8b44e0e4-9e34-4811-a629-0516078a58a1 nodeName:}" failed. No retries permitted until 2026-09-16 01:22:05.385985437 +0000 UTC m=+355.638408004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8b44e0e4-9e34-4811-a629-0516078a58a1-cert") pod "llmisvc-controller-manager-777c6bd7d5-skkpk" (UID: "8b44e0e4-9e34-4811-a629-0516078a58a1") : secret "llmisvc-webhook-server-cert" not found Sep 16 01:22:04.886218 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:22:04.886025 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63ff281e-5d24-44a2-970c-a6ef19b8f64b-cert podName:63ff281e-5d24-44a2-970c-a6ef19b8f64b nodeName:}" failed. No retries permitted until 2026-09-16 01:22:05.386014402 +0000 UTC m=+355.638436991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/63ff281e-5d24-44a2-970c-a6ef19b8f64b-cert") pod "kserve-controller-manager-d6f4b65b8-q42rb" (UID: "63ff281e-5d24-44a2-970c-a6ef19b8f64b") : secret "kserve-webhook-server-cert" not found Sep 16 01:22:04.900711 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.900679 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5kk9\" (UniqueName: \"kubernetes.io/projected/8b44e0e4-9e34-4811-a629-0516078a58a1-kube-api-access-w5kk9\") pod \"llmisvc-controller-manager-777c6bd7d5-skkpk\" (UID: \"8b44e0e4-9e34-4811-a629-0516078a58a1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" Sep 16 01:22:04.900827 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:04.900716 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns7ft\" (UniqueName: \"kubernetes.io/projected/63ff281e-5d24-44a2-970c-a6ef19b8f64b-kube-api-access-ns7ft\") pod \"kserve-controller-manager-d6f4b65b8-q42rb\" (UID: \"63ff281e-5d24-44a2-970c-a6ef19b8f64b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" Sep 16 01:22:05.076169 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.076129 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:05.163821 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.163793 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:05.173588 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.173570 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:22:05.289284 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289255 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-token\") pod \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " Sep 16 01:22:05.289452 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289306 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-data\") pod \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " Sep 16 01:22:05.289452 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289338 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-workload-socket\") pod \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " Sep 16 01:22:05.289452 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289361 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-podinfo\") pod \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " Sep 16 01:22:05.289452 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289447 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65qhb\" (UniqueName: \"kubernetes.io/projected/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-kube-api-access-65qhb\") pod \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " Sep 16 01:22:05.289665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289476 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-credential-socket\") pod \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " Sep 16 01:22:05.289665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289506 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-workload-certs\") pod \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " Sep 16 01:22:05.289665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289533 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istiod-ca-cert\") pod \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " Sep 16 01:22:05.289665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289577 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-envoy\") pod \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\" (UID: \"12b21ef6-f4c1-48d4-a0ab-f524b55ca745\") " Sep 16 01:22:05.289665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289563 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-data" (OuterVolumeSpecName: "istio-data") pod "12b21ef6-f4c1-48d4-a0ab-f524b55ca745" (UID: "12b21ef6-f4c1-48d4-a0ab-f524b55ca745"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:22:05.289665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289633 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "12b21ef6-f4c1-48d4-a0ab-f524b55ca745" (UID: "12b21ef6-f4c1-48d4-a0ab-f524b55ca745"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:22:05.289949 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289770 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "12b21ef6-f4c1-48d4-a0ab-f524b55ca745" (UID: "12b21ef6-f4c1-48d4-a0ab-f524b55ca745"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:22:05.289949 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289819 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "12b21ef6-f4c1-48d4-a0ab-f524b55ca745" (UID: "12b21ef6-f4c1-48d4-a0ab-f524b55ca745"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:22:05.289949 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289872 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-data\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:22:05.289949 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289889 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-workload-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:22:05.289949 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289904 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-workload-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:22:05.289949 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.289887 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "12b21ef6-f4c1-48d4-a0ab-f524b55ca745" (UID: "12b21ef6-f4c1-48d4-a0ab-f524b55ca745"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:22:05.291776 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.291749 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "12b21ef6-f4c1-48d4-a0ab-f524b55ca745" (UID: "12b21ef6-f4c1-48d4-a0ab-f524b55ca745"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:22:05.291888 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.291859 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "12b21ef6-f4c1-48d4-a0ab-f524b55ca745" (UID: "12b21ef6-f4c1-48d4-a0ab-f524b55ca745"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:22:05.291888 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.291873 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-token" (OuterVolumeSpecName: "istio-token") pod "12b21ef6-f4c1-48d4-a0ab-f524b55ca745" (UID: "12b21ef6-f4c1-48d4-a0ab-f524b55ca745"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:22:05.291963 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.291901 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-kube-api-access-65qhb" (OuterVolumeSpecName: "kube-api-access-65qhb") pod "12b21ef6-f4c1-48d4-a0ab-f524b55ca745" (UID: "12b21ef6-f4c1-48d4-a0ab-f524b55ca745"). InnerVolumeSpecName "kube-api-access-65qhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:22:05.390697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.390662 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b44e0e4-9e34-4811-a629-0516078a58a1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-skkpk\" (UID: \"8b44e0e4-9e34-4811-a629-0516078a58a1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" Sep 16 01:22:05.390697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.390693 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63ff281e-5d24-44a2-970c-a6ef19b8f64b-cert\") pod \"kserve-controller-manager-d6f4b65b8-q42rb\" (UID: \"63ff281e-5d24-44a2-970c-a6ef19b8f64b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" Sep 16 01:22:05.390848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.390730 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65qhb\" (UniqueName: \"kubernetes.io/projected/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-kube-api-access-65qhb\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:22:05.390848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.390745 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-credential-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:22:05.390848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.390758 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istiod-ca-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:22:05.390848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.390771 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-envoy\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:22:05.390848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.390783 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-token\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:22:05.390848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.390794 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b21ef6-f4c1-48d4-a0ab-f524b55ca745-istio-podinfo\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:22:05.392911 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.392882 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b44e0e4-9e34-4811-a629-0516078a58a1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-skkpk\" (UID: \"8b44e0e4-9e34-4811-a629-0516078a58a1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" Sep 16 01:22:05.393005 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.392987 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63ff281e-5d24-44a2-970c-a6ef19b8f64b-cert\") pod \"kserve-controller-manager-d6f4b65b8-q42rb\" (UID: \"63ff281e-5d24-44a2-970c-a6ef19b8f64b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" Sep 16 01:22:05.602801 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.602735 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" Sep 16 01:22:05.611686 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.611658 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" Sep 16 01:22:05.888378 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.888347 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q42rb"] Sep 16 01:22:05.891299 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:22:05.891273 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ff281e_5d24_44a2_970c_a6ef19b8f64b.slice/crio-c9e34cd40e3be9fa9516b345d2af96a2bd00570175fb673a33bd1217568c3b58 WatchSource:0}: Error finding container c9e34cd40e3be9fa9516b345d2af96a2bd00570175fb673a33bd1217568c3b58: Status 404 returned error can't find the container with id c9e34cd40e3be9fa9516b345d2af96a2bd00570175fb673a33bd1217568c3b58 Sep 16 01:22:05.909826 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.909804 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk"] Sep 16 01:22:05.911240 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:22:05.911110 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8b44e0e4_9e34_4811_a629_0516078a58a1.slice/crio-3e13f30c32ac53264689d02a3462667f8036445c75fd3148820c31ab6384fd5d WatchSource:0}: Error finding container 3e13f30c32ac53264689d02a3462667f8036445c75fd3148820c31ab6384fd5d: Status 404 returned error can't find the container with id 3e13f30c32ac53264689d02a3462667f8036445c75fd3148820c31ab6384fd5d Sep 16 01:22:05.938242 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.938197 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:05.954110 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.954083 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" event={"ID":"63ff281e-5d24-44a2-970c-a6ef19b8f64b","Type":"ContainerStarted","Data":"c9e34cd40e3be9fa9516b345d2af96a2bd00570175fb673a33bd1217568c3b58"} Sep 16 01:22:05.955377 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.955356 2558 generic.go:356] "Generic (PLEG): container finished" podID="12b21ef6-f4c1-48d4-a0ab-f524b55ca745" containerID="3c2405cee7882fa7852a2f28065498290b6b476a6adf764b7cc1edff0148504a" exitCode=0 Sep 16 01:22:05.955455 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.955402 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" Sep 16 01:22:05.955502 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.955444 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" event={"ID":"12b21ef6-f4c1-48d4-a0ab-f524b55ca745","Type":"ContainerDied","Data":"3c2405cee7882fa7852a2f28065498290b6b476a6adf764b7cc1edff0148504a"} Sep 16 01:22:05.955553 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.955501 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw" event={"ID":"12b21ef6-f4c1-48d4-a0ab-f524b55ca745","Type":"ContainerDied","Data":"12891f8c5b6171a8f527773621f3901b8e8608518936e318e7c6edafc4b5cdac"} Sep 16 01:22:05.955553 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.955536 2558 scope.go:117] "RemoveContainer" containerID="3c2405cee7882fa7852a2f28065498290b6b476a6adf764b7cc1edff0148504a" Sep 16 01:22:05.956801 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.956780 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" event={"ID":"8b44e0e4-9e34-4811-a629-0516078a58a1","Type":"ContainerStarted","Data":"3e13f30c32ac53264689d02a3462667f8036445c75fd3148820c31ab6384fd5d"} Sep 16 01:22:05.964531 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.964518 2558 scope.go:117] "RemoveContainer" containerID="3c2405cee7882fa7852a2f28065498290b6b476a6adf764b7cc1edff0148504a" Sep 16 01:22:05.964789 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:22:05.964770 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2405cee7882fa7852a2f28065498290b6b476a6adf764b7cc1edff0148504a\": container with ID starting with 3c2405cee7882fa7852a2f28065498290b6b476a6adf764b7cc1edff0148504a not found: ID does not exist" containerID="3c2405cee7882fa7852a2f28065498290b6b476a6adf764b7cc1edff0148504a" Sep 16 01:22:05.964837 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.964795 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2405cee7882fa7852a2f28065498290b6b476a6adf764b7cc1edff0148504a"} err="failed to get container status \"3c2405cee7882fa7852a2f28065498290b6b476a6adf764b7cc1edff0148504a\": rpc error: code = NotFound desc = could not find container \"3c2405cee7882fa7852a2f28065498290b6b476a6adf764b7cc1edff0148504a\": container with ID starting with 3c2405cee7882fa7852a2f28065498290b6b476a6adf764b7cc1edff0148504a not found: ID does not exist" Sep 16 01:22:05.977393 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.977372 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw"] Sep 16 01:22:05.982970 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:05.982949 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfl5tw"] Sep 16 01:22:06.004338 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:06.004322 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:06.063858 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:06.063834 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:06.294799 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:06.294735 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12b21ef6-f4c1-48d4-a0ab-f524b55ca745" path="/var/lib/kubelet/pods/12b21ef6-f4c1-48d4-a0ab-f524b55ca745/volumes" Sep 16 01:22:08.977377 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:08.977346 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" event={"ID":"63ff281e-5d24-44a2-970c-a6ef19b8f64b","Type":"ContainerStarted","Data":"4aac64685d388ad80292e57a358f460891d7c326d38076995024644081eb09f4"} Sep 16 01:22:08.977680 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:08.977475 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" Sep 16 01:22:08.994336 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:08.994296 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" podStartSLOduration=2.006393829 podStartE2EDuration="4.994282996s" totalImagesPullingTime="2.987889167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:22:08.993569872 +0000 UTC m=+359.245992463" watchObservedRunningTime="2026-09-16 01:22:08.994282996 +0000 UTC m=+359.246705585" Sep 16 01:22:09.983574 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:09.983535 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" event={"ID":"8b44e0e4-9e34-4811-a629-0516078a58a1","Type":"ContainerStarted","Data":"24ebe92647134775a5712f13ce759f09df85d50b31d396348c14576a587bac64"} Sep 16 01:22:10.000002 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:09.999946 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" podStartSLOduration=2.975070611 podStartE2EDuration="5.9999305s" totalImagesPullingTime="3.024859889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:22:09.99786667 +0000 UTC m=+360.250289260" watchObservedRunningTime="2026-09-16 01:22:09.9999305 +0000 UTC m=+360.252353141" Sep 16 01:22:10.987857 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:10.987829 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" Sep 16 01:22:12.940831 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:12.940801 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lkdhm" Sep 16 01:22:12.942566 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:12.942542 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mpg47" Sep 16 01:22:16.509880 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:16.509839 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:16.562899 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:16.562868 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:17.996429 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:17.996388 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:18.048020 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:18.047994 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:27.601312 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.601279 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c8c848f8f-plcg9"] Sep 16 01:22:27.602080 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.602040 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b21ef6-f4c1-48d4-a0ab-f524b55ca745" containerName="istio-proxy" Sep 16 01:22:27.602080 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.602064 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b21ef6-f4c1-48d4-a0ab-f524b55ca745" containerName="istio-proxy" Sep 16 01:22:27.602333 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.602190 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b21ef6-f4c1-48d4-a0ab-f524b55ca745" containerName="istio-proxy" Sep 16 01:22:27.610540 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.610521 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.614899 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.614876 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c8c848f8f-plcg9"] Sep 16 01:22:27.652447 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.652415 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:27.662490 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.662464 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d114a491-470a-408f-b155-e8ae4d514341-console-serving-cert\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.662596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.662501 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d114a491-470a-408f-b155-e8ae4d514341-trusted-ca-bundle\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.662596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.662528 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9jvv\" (UniqueName: \"kubernetes.io/projected/d114a491-470a-408f-b155-e8ae4d514341-kube-api-access-m9jvv\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.662596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.662581 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d114a491-470a-408f-b155-e8ae4d514341-oauth-serving-cert\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.662763 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.662612 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d114a491-470a-408f-b155-e8ae4d514341-service-ca\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.662763 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.662656 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d114a491-470a-408f-b155-e8ae4d514341-console-config\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.662763 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.662679 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d114a491-470a-408f-b155-e8ae4d514341-console-oauth-config\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.753552 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.753519 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:27.763579 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.763556 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d114a491-470a-408f-b155-e8ae4d514341-console-serving-cert\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.763663 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.763585 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d114a491-470a-408f-b155-e8ae4d514341-trusted-ca-bundle\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.763701 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.763671 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jvv\" (UniqueName: \"kubernetes.io/projected/d114a491-470a-408f-b155-e8ae4d514341-kube-api-access-m9jvv\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.763744 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.763730 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d114a491-470a-408f-b155-e8ae4d514341-oauth-serving-cert\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.763785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.763764 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d114a491-470a-408f-b155-e8ae4d514341-service-ca\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.763837 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.763809 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d114a491-470a-408f-b155-e8ae4d514341-console-config\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.763837 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.763830 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d114a491-470a-408f-b155-e8ae4d514341-console-oauth-config\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.764529 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.764505 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d114a491-470a-408f-b155-e8ae4d514341-trusted-ca-bundle\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.764627 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.764521 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d114a491-470a-408f-b155-e8ae4d514341-oauth-serving-cert\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.764696 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.764575 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d114a491-470a-408f-b155-e8ae4d514341-service-ca\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.764869 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.764843 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d114a491-470a-408f-b155-e8ae4d514341-console-config\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.766437 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.766418 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d114a491-470a-408f-b155-e8ae4d514341-console-serving-cert\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.766523 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.766486 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d114a491-470a-408f-b155-e8ae4d514341-console-oauth-config\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.771334 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.771308 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jvv\" (UniqueName: \"kubernetes.io/projected/d114a491-470a-408f-b155-e8ae4d514341-kube-api-access-m9jvv\") pod \"console-5c8c848f8f-plcg9\" (UID: \"d114a491-470a-408f-b155-e8ae4d514341\") " pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:27.854272 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.854209 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:27.921629 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:27.921604 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:28.197822 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:28.197782 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c8c848f8f-plcg9"] Sep 16 01:22:28.203056 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:22:28.203013 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd114a491_470a_408f_b155_e8ae4d514341.slice/crio-27ef3abbc1624719c583494be7b011b90fec137ffa6c75453b1fe285c2883796 WatchSource:0}: Error finding container 27ef3abbc1624719c583494be7b011b90fec137ffa6c75453b1fe285c2883796: Status 404 returned error can't find the container with id 27ef3abbc1624719c583494be7b011b90fec137ffa6c75453b1fe285c2883796 Sep 16 01:22:28.396931 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:28.396897 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:29.075638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:29.075599 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8c848f8f-plcg9" event={"ID":"d114a491-470a-408f-b155-e8ae4d514341","Type":"ContainerStarted","Data":"d66ccbf7bef0a69589ac979d3324312699d78efced3d89d6bea96d90faa6effc"} Sep 16 01:22:29.075979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:29.075657 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8c848f8f-plcg9" event={"ID":"d114a491-470a-408f-b155-e8ae4d514341","Type":"ContainerStarted","Data":"27ef3abbc1624719c583494be7b011b90fec137ffa6c75453b1fe285c2883796"} Sep 16 01:22:29.091720 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:29.091668 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c8c848f8f-plcg9" podStartSLOduration=2.091652502 podStartE2EDuration="2.091652502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:22:29.090270017 +0000 UTC m=+379.342692605" watchObservedRunningTime="2026-09-16 01:22:29.091652502 +0000 UTC m=+379.344075090" Sep 16 01:22:37.922526 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:37.922485 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:37.922893 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:37.922559 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:37.927417 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:37.927394 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:38.122506 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:38.122479 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c8c848f8f-plcg9" Sep 16 01:22:38.163504 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:38.163463 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67b955f76d-5566h"] Sep 16 01:22:38.262290 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:38.262209 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:38.362700 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:38.362664 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:38.411950 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:38.411921 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:39.988791 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:39.988750 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q42rb" Sep 16 01:22:41.994660 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:41.994633 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-skkpk" Sep 16 01:22:56.615065 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:56.615030 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:56.717030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:56.717001 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:22:59.914601 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:59.914563 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz"] Sep 16 01:22:59.925806 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:59.925780 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz"] Sep 16 01:22:59.925958 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:59.925877 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" Sep 16 01:22:59.928327 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:59.928298 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 01:22:59.928477 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:59.928298 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 01:22:59.928942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:22:59.928922 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hgbd4\"" Sep 16 01:23:00.006358 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.006332 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2vk5\" (UniqueName: \"kubernetes.io/projected/937ea01c-5360-456b-8b4b-fd8271bc0283-kube-api-access-f2vk5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz\" (UID: \"937ea01c-5360-456b-8b4b-fd8271bc0283\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" Sep 16 01:23:00.006489 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.006382 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937ea01c-5360-456b-8b4b-fd8271bc0283-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz\" (UID: \"937ea01c-5360-456b-8b4b-fd8271bc0283\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" Sep 16 01:23:00.006489 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.006467 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937ea01c-5360-456b-8b4b-fd8271bc0283-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz\" (UID: \"937ea01c-5360-456b-8b4b-fd8271bc0283\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" Sep 16 01:23:00.011478 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.011456 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b"] Sep 16 01:23:00.014903 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.014888 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" Sep 16 01:23:00.023561 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.023537 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b"] Sep 16 01:23:00.107100 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.107069 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937ea01c-5360-456b-8b4b-fd8271bc0283-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz\" (UID: \"937ea01c-5360-456b-8b4b-fd8271bc0283\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" Sep 16 01:23:00.107269 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.107124 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vk5\" (UniqueName: \"kubernetes.io/projected/937ea01c-5360-456b-8b4b-fd8271bc0283-kube-api-access-f2vk5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz\" (UID: \"937ea01c-5360-456b-8b4b-fd8271bc0283\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" Sep 16 01:23:00.107269 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.107183 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937ea01c-5360-456b-8b4b-fd8271bc0283-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz\" (UID: \"937ea01c-5360-456b-8b4b-fd8271bc0283\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" Sep 16 01:23:00.107269 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.107215 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2738cf67-615e-4981-af7b-ef54f58693e1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b\" (UID: \"2738cf67-615e-4981-af7b-ef54f58693e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" Sep 16 01:23:00.107269 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.107253 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9xm9\" (UniqueName: \"kubernetes.io/projected/2738cf67-615e-4981-af7b-ef54f58693e1-kube-api-access-k9xm9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b\" (UID: \"2738cf67-615e-4981-af7b-ef54f58693e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" Sep 16 01:23:00.107496 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.107293 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2738cf67-615e-4981-af7b-ef54f58693e1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b\" (UID: \"2738cf67-615e-4981-af7b-ef54f58693e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" Sep 16 01:23:00.107553 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.107504 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937ea01c-5360-456b-8b4b-fd8271bc0283-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz\" (UID: \"937ea01c-5360-456b-8b4b-fd8271bc0283\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" Sep 16 01:23:00.107604 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.107548 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937ea01c-5360-456b-8b4b-fd8271bc0283-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz\" (UID: \"937ea01c-5360-456b-8b4b-fd8271bc0283\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" Sep 16 01:23:00.111868 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.111847 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx"] Sep 16 01:23:00.114469 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.114442 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vk5\" (UniqueName: \"kubernetes.io/projected/937ea01c-5360-456b-8b4b-fd8271bc0283-kube-api-access-f2vk5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz\" (UID: \"937ea01c-5360-456b-8b4b-fd8271bc0283\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" Sep 16 01:23:00.115360 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.115346 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" Sep 16 01:23:00.122120 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.122099 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx"] Sep 16 01:23:00.208527 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.208449 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3eab58e-4cfa-4f2e-a440-2b998d95a384-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx\" (UID: \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" Sep 16 01:23:00.208527 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.208497 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2738cf67-615e-4981-af7b-ef54f58693e1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b\" (UID: \"2738cf67-615e-4981-af7b-ef54f58693e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" Sep 16 01:23:00.208527 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.208524 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9xm9\" (UniqueName: \"kubernetes.io/projected/2738cf67-615e-4981-af7b-ef54f58693e1-kube-api-access-k9xm9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b\" (UID: \"2738cf67-615e-4981-af7b-ef54f58693e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" Sep 16 01:23:00.208803 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.208560 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m45ng\" (UniqueName: \"kubernetes.io/projected/f3eab58e-4cfa-4f2e-a440-2b998d95a384-kube-api-access-m45ng\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx\" (UID: \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" Sep 16 01:23:00.208803 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.208578 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2738cf67-615e-4981-af7b-ef54f58693e1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b\" (UID: \"2738cf67-615e-4981-af7b-ef54f58693e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" Sep 16 01:23:00.208803 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.208658 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3eab58e-4cfa-4f2e-a440-2b998d95a384-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx\" (UID: \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" Sep 16 01:23:00.208967 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.208902 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2738cf67-615e-4981-af7b-ef54f58693e1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b\" (UID: \"2738cf67-615e-4981-af7b-ef54f58693e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" Sep 16 01:23:00.208967 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.208900 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2738cf67-615e-4981-af7b-ef54f58693e1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b\" (UID: \"2738cf67-615e-4981-af7b-ef54f58693e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" Sep 16 01:23:00.211607 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.211583 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd"] Sep 16 01:23:00.215111 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.215094 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" Sep 16 01:23:00.216332 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.216311 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9xm9\" (UniqueName: \"kubernetes.io/projected/2738cf67-615e-4981-af7b-ef54f58693e1-kube-api-access-k9xm9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b\" (UID: \"2738cf67-615e-4981-af7b-ef54f58693e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" Sep 16 01:23:00.222469 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.222446 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd"] Sep 16 01:23:00.236621 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.236601 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" Sep 16 01:23:00.309266 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.309130 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3eab58e-4cfa-4f2e-a440-2b998d95a384-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx\" (UID: \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" Sep 16 01:23:00.309266 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.309213 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd\" (UID: \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" Sep 16 01:23:00.309481 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.309268 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3eab58e-4cfa-4f2e-a440-2b998d95a384-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx\" (UID: \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" Sep 16 01:23:00.309481 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.309344 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlhxw\" (UniqueName: \"kubernetes.io/projected/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-kube-api-access-zlhxw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd\" (UID: \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" Sep 16 01:23:00.309481 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.309393 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m45ng\" (UniqueName: \"kubernetes.io/projected/f3eab58e-4cfa-4f2e-a440-2b998d95a384-kube-api-access-m45ng\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx\" (UID: \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" Sep 16 01:23:00.309631 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.309481 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd\" (UID: \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" Sep 16 01:23:00.309631 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.309524 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3eab58e-4cfa-4f2e-a440-2b998d95a384-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx\" (UID: \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" Sep 16 01:23:00.309631 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.309570 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3eab58e-4cfa-4f2e-a440-2b998d95a384-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx\" (UID: \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" Sep 16 01:23:00.317658 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.317622 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m45ng\" (UniqueName: \"kubernetes.io/projected/f3eab58e-4cfa-4f2e-a440-2b998d95a384-kube-api-access-m45ng\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx\" (UID: \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" Sep 16 01:23:00.323732 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.323710 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" Sep 16 01:23:00.410789 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.410760 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd\" (UID: \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" Sep 16 01:23:00.410941 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.410857 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd\" (UID: \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" Sep 16 01:23:00.411027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.410976 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlhxw\" (UniqueName: \"kubernetes.io/projected/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-kube-api-access-zlhxw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd\" (UID: \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" Sep 16 01:23:00.411244 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.411223 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd\" (UID: \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" Sep 16 01:23:00.411383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.411364 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd\" (UID: \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" Sep 16 01:23:00.419417 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.419398 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlhxw\" (UniqueName: \"kubernetes.io/projected/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-kube-api-access-zlhxw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd\" (UID: \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" Sep 16 01:23:00.431318 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.431292 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" Sep 16 01:23:00.510001 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.509976 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz"] Sep 16 01:23:00.511530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:23:00.511501 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937ea01c_5360_456b_8b4b_fd8271bc0283.slice/crio-5210128c19296175752de2dd7f238f7c17cc02eddea16dfca0d8e806fad06dd3 WatchSource:0}: Error finding container 5210128c19296175752de2dd7f238f7c17cc02eddea16dfca0d8e806fad06dd3: Status 404 returned error can't find the container with id 5210128c19296175752de2dd7f238f7c17cc02eddea16dfca0d8e806fad06dd3 Sep 16 01:23:00.533775 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.533747 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" Sep 16 01:23:00.598293 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.598266 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b"] Sep 16 01:23:00.600454 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:23:00.600405 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2738cf67_615e_4981_af7b_ef54f58693e1.slice/crio-568729c698a4f7a52f14ab45e01ac4a6117a2f967fd17e598abf5191b19b8984 WatchSource:0}: Error finding container 568729c698a4f7a52f14ab45e01ac4a6117a2f967fd17e598abf5191b19b8984: Status 404 returned error can't find the container with id 568729c698a4f7a52f14ab45e01ac4a6117a2f967fd17e598abf5191b19b8984 Sep 16 01:23:00.701545 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.701520 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx"] Sep 16 01:23:00.715204 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:23:00.715148 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3eab58e_4cfa_4f2e_a440_2b998d95a384.slice/crio-c1197d872430720cf302afb07e2ae83acd4100e3288929834b3590d664bb0753 WatchSource:0}: Error finding container c1197d872430720cf302afb07e2ae83acd4100e3288929834b3590d664bb0753: Status 404 returned error can't find the container with id c1197d872430720cf302afb07e2ae83acd4100e3288929834b3590d664bb0753 Sep 16 01:23:00.809750 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:00.808216 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd"] Sep 16 01:23:00.860927 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:23:00.860899 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0834b1_bb95_4ad2_8a14_f66d2dbb77d3.slice/crio-75d6bbc8d310c1d95fb31d24105d41f3a77ef4fdcb3b240644ca495e82ed3b46 WatchSource:0}: Error finding container 75d6bbc8d310c1d95fb31d24105d41f3a77ef4fdcb3b240644ca495e82ed3b46: Status 404 returned error can't find the container with id 75d6bbc8d310c1d95fb31d24105d41f3a77ef4fdcb3b240644ca495e82ed3b46 Sep 16 01:23:01.240727 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:01.240698 2558 generic.go:356] "Generic (PLEG): container finished" podID="f3eab58e-4cfa-4f2e-a440-2b998d95a384" containerID="5add0de877c38d2f270caa0a8f2dd7f3ef25993a24163131b8696150b95fa257" exitCode=0 Sep 16 01:23:01.241153 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:01.240780 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" event={"ID":"f3eab58e-4cfa-4f2e-a440-2b998d95a384","Type":"ContainerDied","Data":"5add0de877c38d2f270caa0a8f2dd7f3ef25993a24163131b8696150b95fa257"} Sep 16 01:23:01.241153 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:01.240830 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" event={"ID":"f3eab58e-4cfa-4f2e-a440-2b998d95a384","Type":"ContainerStarted","Data":"c1197d872430720cf302afb07e2ae83acd4100e3288929834b3590d664bb0753"} Sep 16 01:23:01.242395 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:01.242375 2558 generic.go:356] "Generic (PLEG): container finished" podID="937ea01c-5360-456b-8b4b-fd8271bc0283" containerID="6b6f16d4c92a26f8563cf452a6515521fb70477ca95d308f16610676941599a5" exitCode=0 Sep 16 01:23:01.242486 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:01.242463 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" event={"ID":"937ea01c-5360-456b-8b4b-fd8271bc0283","Type":"ContainerDied","Data":"6b6f16d4c92a26f8563cf452a6515521fb70477ca95d308f16610676941599a5"} Sep 16 01:23:01.242545 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:01.242515 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" event={"ID":"937ea01c-5360-456b-8b4b-fd8271bc0283","Type":"ContainerStarted","Data":"5210128c19296175752de2dd7f238f7c17cc02eddea16dfca0d8e806fad06dd3"} Sep 16 01:23:01.244240 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:01.244209 2558 generic.go:356] "Generic (PLEG): container finished" podID="6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3" containerID="bb610f39de6e19cee972eb7451c243bdae8f078aaa0169e4bd8d3af0454caa55" exitCode=0 Sep 16 01:23:01.244311 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:01.244283 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" event={"ID":"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3","Type":"ContainerDied","Data":"bb610f39de6e19cee972eb7451c243bdae8f078aaa0169e4bd8d3af0454caa55"} Sep 16 01:23:01.244358 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:01.244320 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" event={"ID":"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3","Type":"ContainerStarted","Data":"75d6bbc8d310c1d95fb31d24105d41f3a77ef4fdcb3b240644ca495e82ed3b46"} Sep 16 01:23:01.246073 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:01.246054 2558 generic.go:356] "Generic (PLEG): container finished" podID="2738cf67-615e-4981-af7b-ef54f58693e1" containerID="9027846a28ad14882b9bde077716cc4821c562a6f1182b60e83fec3b82dd9809" exitCode=0 Sep 16 01:23:01.246190 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:01.246123 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" event={"ID":"2738cf67-615e-4981-af7b-ef54f58693e1","Type":"ContainerDied","Data":"9027846a28ad14882b9bde077716cc4821c562a6f1182b60e83fec3b82dd9809"} Sep 16 01:23:01.246265 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:01.246209 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" event={"ID":"2738cf67-615e-4981-af7b-ef54f58693e1","Type":"ContainerStarted","Data":"568729c698a4f7a52f14ab45e01ac4a6117a2f967fd17e598abf5191b19b8984"} Sep 16 01:23:03.185965 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.185924 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67b955f76d-5566h" podUID="13bba4d5-c971-40af-b470-52b17914a115" containerName="console" containerID="cri-o://1c0dfc7799ad9b307166a96419f4a42b930b065dc854ff8c18c273a2666f7fbd" gracePeriod=15 Sep 16 01:23:03.257611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.257583 2558 generic.go:356] "Generic (PLEG): container finished" podID="2738cf67-615e-4981-af7b-ef54f58693e1" containerID="822386b421c143ec1fab05d3bdc317a0b48104981b9263973237a3c3812962bb" exitCode=0 Sep 16 01:23:03.257741 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.257673 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" event={"ID":"2738cf67-615e-4981-af7b-ef54f58693e1","Type":"ContainerDied","Data":"822386b421c143ec1fab05d3bdc317a0b48104981b9263973237a3c3812962bb"} Sep 16 01:23:03.259718 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.259699 2558 generic.go:356] "Generic (PLEG): container finished" podID="f3eab58e-4cfa-4f2e-a440-2b998d95a384" containerID="ab9b2b852d640de50403432d18d6e6d4322ae8bb9a4321d5c23afa5ab4eec207" exitCode=0 Sep 16 01:23:03.259804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.259759 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" event={"ID":"f3eab58e-4cfa-4f2e-a440-2b998d95a384","Type":"ContainerDied","Data":"ab9b2b852d640de50403432d18d6e6d4322ae8bb9a4321d5c23afa5ab4eec207"} Sep 16 01:23:03.261869 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.261850 2558 generic.go:356] "Generic (PLEG): container finished" podID="937ea01c-5360-456b-8b4b-fd8271bc0283" containerID="402ebbf2a5047019f81b8da19b7e12f3e02b254efa08d90a446c670fdd48bce4" exitCode=0 Sep 16 01:23:03.261955 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.261878 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" event={"ID":"937ea01c-5360-456b-8b4b-fd8271bc0283","Type":"ContainerDied","Data":"402ebbf2a5047019f81b8da19b7e12f3e02b254efa08d90a446c670fdd48bce4"} Sep 16 01:23:03.579308 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.579283 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:23:03.739572 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.739548 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13bba4d5-c971-40af-b470-52b17914a115-console-serving-cert\") pod \"13bba4d5-c971-40af-b470-52b17914a115\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " Sep 16 01:23:03.739690 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.739593 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-trusted-ca-bundle\") pod \"13bba4d5-c971-40af-b470-52b17914a115\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " Sep 16 01:23:03.739690 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.739618 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-console-config\") pod \"13bba4d5-c971-40af-b470-52b17914a115\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " Sep 16 01:23:03.739690 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.739674 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13bba4d5-c971-40af-b470-52b17914a115-console-oauth-config\") pod \"13bba4d5-c971-40af-b470-52b17914a115\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " Sep 16 01:23:03.739861 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.739808 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns5tz\" (UniqueName: \"kubernetes.io/projected/13bba4d5-c971-40af-b470-52b17914a115-kube-api-access-ns5tz\") pod \"13bba4d5-c971-40af-b470-52b17914a115\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " Sep 16 01:23:03.739923 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.739886 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-service-ca\") pod \"13bba4d5-c971-40af-b470-52b17914a115\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " Sep 16 01:23:03.739975 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.739922 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-oauth-serving-cert\") pod \"13bba4d5-c971-40af-b470-52b17914a115\" (UID: \"13bba4d5-c971-40af-b470-52b17914a115\") " Sep 16 01:23:03.740044 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.740009 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-console-config" (OuterVolumeSpecName: "console-config") pod "13bba4d5-c971-40af-b470-52b17914a115" (UID: "13bba4d5-c971-40af-b470-52b17914a115"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:23:03.740102 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.740054 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "13bba4d5-c971-40af-b470-52b17914a115" (UID: "13bba4d5-c971-40af-b470-52b17914a115"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:23:03.740311 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.740255 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-trusted-ca-bundle\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:03.740311 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.740272 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-service-ca" (OuterVolumeSpecName: "service-ca") pod "13bba4d5-c971-40af-b470-52b17914a115" (UID: "13bba4d5-c971-40af-b470-52b17914a115"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:23:03.740311 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.740284 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "13bba4d5-c971-40af-b470-52b17914a115" (UID: "13bba4d5-c971-40af-b470-52b17914a115"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:23:03.740311 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.740279 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-console-config\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:03.741818 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.741800 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13bba4d5-c971-40af-b470-52b17914a115-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "13bba4d5-c971-40af-b470-52b17914a115" (UID: "13bba4d5-c971-40af-b470-52b17914a115"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:23:03.741902 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.741830 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13bba4d5-c971-40af-b470-52b17914a115-kube-api-access-ns5tz" (OuterVolumeSpecName: "kube-api-access-ns5tz") pod "13bba4d5-c971-40af-b470-52b17914a115" (UID: "13bba4d5-c971-40af-b470-52b17914a115"). InnerVolumeSpecName "kube-api-access-ns5tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:23:03.741992 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.741977 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13bba4d5-c971-40af-b470-52b17914a115-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "13bba4d5-c971-40af-b470-52b17914a115" (UID: "13bba4d5-c971-40af-b470-52b17914a115"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:23:03.841618 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.841591 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns5tz\" (UniqueName: \"kubernetes.io/projected/13bba4d5-c971-40af-b470-52b17914a115-kube-api-access-ns5tz\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:03.841618 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.841615 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-service-ca\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:03.841767 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.841625 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13bba4d5-c971-40af-b470-52b17914a115-oauth-serving-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:03.841767 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.841635 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13bba4d5-c971-40af-b470-52b17914a115-console-serving-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:03.841767 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:03.841643 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13bba4d5-c971-40af-b470-52b17914a115-console-oauth-config\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:04.268973 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.268942 2558 generic.go:356] "Generic (PLEG): container finished" podID="6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3" containerID="ea6db61124ae2639f101d8c03542c687463e9659dbf2d7d0ef9a891fa7c9a6c9" exitCode=0 Sep 16 01:23:04.269386 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.269036 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" event={"ID":"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3","Type":"ContainerDied","Data":"ea6db61124ae2639f101d8c03542c687463e9659dbf2d7d0ef9a891fa7c9a6c9"} Sep 16 01:23:04.271420 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.271399 2558 generic.go:356] "Generic (PLEG): container finished" podID="2738cf67-615e-4981-af7b-ef54f58693e1" containerID="a309976b8c86d71c11f03f7927ce315d5454ae80aa5475fc86418ad5a5c1ac24" exitCode=0 Sep 16 01:23:04.271548 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.271527 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" event={"ID":"2738cf67-615e-4981-af7b-ef54f58693e1","Type":"ContainerDied","Data":"a309976b8c86d71c11f03f7927ce315d5454ae80aa5475fc86418ad5a5c1ac24"} Sep 16 01:23:04.272749 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.272727 2558 generic.go:356] "Generic (PLEG): container finished" podID="13bba4d5-c971-40af-b470-52b17914a115" containerID="1c0dfc7799ad9b307166a96419f4a42b930b065dc854ff8c18c273a2666f7fbd" exitCode=0 Sep 16 01:23:04.272844 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.272780 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b955f76d-5566h" Sep 16 01:23:04.272844 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.272807 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b955f76d-5566h" event={"ID":"13bba4d5-c971-40af-b470-52b17914a115","Type":"ContainerDied","Data":"1c0dfc7799ad9b307166a96419f4a42b930b065dc854ff8c18c273a2666f7fbd"} Sep 16 01:23:04.272971 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.272849 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b955f76d-5566h" event={"ID":"13bba4d5-c971-40af-b470-52b17914a115","Type":"ContainerDied","Data":"4ef31d7f26be6dea95f716faad5e68e0281b8f2341e373c4d2ce3142b15c4410"} Sep 16 01:23:04.272971 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.272872 2558 scope.go:117] "RemoveContainer" containerID="1c0dfc7799ad9b307166a96419f4a42b930b065dc854ff8c18c273a2666f7fbd" Sep 16 01:23:04.275433 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.275376 2558 generic.go:356] "Generic (PLEG): container finished" podID="f3eab58e-4cfa-4f2e-a440-2b998d95a384" containerID="2529a6cd4931981840a308d01e87ee875d4cd0a2176eecc8519613a437e716a9" exitCode=0 Sep 16 01:23:04.275433 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.275407 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" event={"ID":"f3eab58e-4cfa-4f2e-a440-2b998d95a384","Type":"ContainerDied","Data":"2529a6cd4931981840a308d01e87ee875d4cd0a2176eecc8519613a437e716a9"} Sep 16 01:23:04.277977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.277955 2558 generic.go:356] "Generic (PLEG): container finished" podID="937ea01c-5360-456b-8b4b-fd8271bc0283" containerID="eab1c50d30d1c0f01f0e27578bd69ee4179765b7794e741cf63616e300f53ddb" exitCode=0 Sep 16 01:23:04.278062 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.278033 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" event={"ID":"937ea01c-5360-456b-8b4b-fd8271bc0283","Type":"ContainerDied","Data":"eab1c50d30d1c0f01f0e27578bd69ee4179765b7794e741cf63616e300f53ddb"} Sep 16 01:23:04.299540 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.299520 2558 scope.go:117] "RemoveContainer" containerID="1c0dfc7799ad9b307166a96419f4a42b930b065dc854ff8c18c273a2666f7fbd" Sep 16 01:23:04.299826 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:23:04.299808 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0dfc7799ad9b307166a96419f4a42b930b065dc854ff8c18c273a2666f7fbd\": container with ID starting with 1c0dfc7799ad9b307166a96419f4a42b930b065dc854ff8c18c273a2666f7fbd not found: ID does not exist" containerID="1c0dfc7799ad9b307166a96419f4a42b930b065dc854ff8c18c273a2666f7fbd" Sep 16 01:23:04.299880 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.299834 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0dfc7799ad9b307166a96419f4a42b930b065dc854ff8c18c273a2666f7fbd"} err="failed to get container status \"1c0dfc7799ad9b307166a96419f4a42b930b065dc854ff8c18c273a2666f7fbd\": rpc error: code = NotFound desc = could not find container \"1c0dfc7799ad9b307166a96419f4a42b930b065dc854ff8c18c273a2666f7fbd\": container with ID starting with 1c0dfc7799ad9b307166a96419f4a42b930b065dc854ff8c18c273a2666f7fbd not found: ID does not exist" Sep 16 01:23:04.344544 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.344521 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67b955f76d-5566h"] Sep 16 01:23:04.350533 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:04.350509 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67b955f76d-5566h"] Sep 16 01:23:05.286111 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.286080 2558 generic.go:356] "Generic (PLEG): container finished" podID="6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3" containerID="0009743f6027602077eaa7aa1e4b9fcffc163c23ecca14f543a91fd332820a13" exitCode=0 Sep 16 01:23:05.286514 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.286191 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" event={"ID":"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3","Type":"ContainerDied","Data":"0009743f6027602077eaa7aa1e4b9fcffc163c23ecca14f543a91fd332820a13"} Sep 16 01:23:05.431801 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.431779 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" Sep 16 01:23:05.471760 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.471731 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" Sep 16 01:23:05.507546 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.507524 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" Sep 16 01:23:05.555653 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.555590 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2738cf67-615e-4981-af7b-ef54f58693e1-bundle\") pod \"2738cf67-615e-4981-af7b-ef54f58693e1\" (UID: \"2738cf67-615e-4981-af7b-ef54f58693e1\") " Sep 16 01:23:05.555653 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.555648 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937ea01c-5360-456b-8b4b-fd8271bc0283-util\") pod \"937ea01c-5360-456b-8b4b-fd8271bc0283\" (UID: \"937ea01c-5360-456b-8b4b-fd8271bc0283\") " Sep 16 01:23:05.555804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.555670 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2738cf67-615e-4981-af7b-ef54f58693e1-util\") pod \"2738cf67-615e-4981-af7b-ef54f58693e1\" (UID: \"2738cf67-615e-4981-af7b-ef54f58693e1\") " Sep 16 01:23:05.555804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.555737 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2vk5\" (UniqueName: \"kubernetes.io/projected/937ea01c-5360-456b-8b4b-fd8271bc0283-kube-api-access-f2vk5\") pod \"937ea01c-5360-456b-8b4b-fd8271bc0283\" (UID: \"937ea01c-5360-456b-8b4b-fd8271bc0283\") " Sep 16 01:23:05.555804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.555760 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9xm9\" (UniqueName: \"kubernetes.io/projected/2738cf67-615e-4981-af7b-ef54f58693e1-kube-api-access-k9xm9\") pod \"2738cf67-615e-4981-af7b-ef54f58693e1\" (UID: \"2738cf67-615e-4981-af7b-ef54f58693e1\") " Sep 16 01:23:05.555969 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.555814 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937ea01c-5360-456b-8b4b-fd8271bc0283-bundle\") pod \"937ea01c-5360-456b-8b4b-fd8271bc0283\" (UID: \"937ea01c-5360-456b-8b4b-fd8271bc0283\") " Sep 16 01:23:05.556642 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.556402 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937ea01c-5360-456b-8b4b-fd8271bc0283-bundle" (OuterVolumeSpecName: "bundle") pod "937ea01c-5360-456b-8b4b-fd8271bc0283" (UID: "937ea01c-5360-456b-8b4b-fd8271bc0283"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:23:05.556642 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.556431 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2738cf67-615e-4981-af7b-ef54f58693e1-bundle" (OuterVolumeSpecName: "bundle") pod "2738cf67-615e-4981-af7b-ef54f58693e1" (UID: "2738cf67-615e-4981-af7b-ef54f58693e1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:23:05.557998 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.557976 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937ea01c-5360-456b-8b4b-fd8271bc0283-kube-api-access-f2vk5" (OuterVolumeSpecName: "kube-api-access-f2vk5") pod "937ea01c-5360-456b-8b4b-fd8271bc0283" (UID: "937ea01c-5360-456b-8b4b-fd8271bc0283"). InnerVolumeSpecName "kube-api-access-f2vk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:23:05.558071 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.558037 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2738cf67-615e-4981-af7b-ef54f58693e1-kube-api-access-k9xm9" (OuterVolumeSpecName: "kube-api-access-k9xm9") pod "2738cf67-615e-4981-af7b-ef54f58693e1" (UID: "2738cf67-615e-4981-af7b-ef54f58693e1"). InnerVolumeSpecName "kube-api-access-k9xm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:23:05.561317 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.561296 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2738cf67-615e-4981-af7b-ef54f58693e1-util" (OuterVolumeSpecName: "util") pod "2738cf67-615e-4981-af7b-ef54f58693e1" (UID: "2738cf67-615e-4981-af7b-ef54f58693e1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:23:05.561889 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.561870 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937ea01c-5360-456b-8b4b-fd8271bc0283-util" (OuterVolumeSpecName: "util") pod "937ea01c-5360-456b-8b4b-fd8271bc0283" (UID: "937ea01c-5360-456b-8b4b-fd8271bc0283"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:23:05.657137 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.657114 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3eab58e-4cfa-4f2e-a440-2b998d95a384-bundle\") pod \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\" (UID: \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\") " Sep 16 01:23:05.657260 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.657190 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3eab58e-4cfa-4f2e-a440-2b998d95a384-util\") pod \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\" (UID: \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\") " Sep 16 01:23:05.657260 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.657210 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m45ng\" (UniqueName: \"kubernetes.io/projected/f3eab58e-4cfa-4f2e-a440-2b998d95a384-kube-api-access-m45ng\") pod \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\" (UID: \"f3eab58e-4cfa-4f2e-a440-2b998d95a384\") " Sep 16 01:23:05.657366 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.657355 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2vk5\" (UniqueName: \"kubernetes.io/projected/937ea01c-5360-456b-8b4b-fd8271bc0283-kube-api-access-f2vk5\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:05.657400 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.657370 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9xm9\" (UniqueName: \"kubernetes.io/projected/2738cf67-615e-4981-af7b-ef54f58693e1-kube-api-access-k9xm9\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:05.657400 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.657380 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937ea01c-5360-456b-8b4b-fd8271bc0283-bundle\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:05.657400 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.657389 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2738cf67-615e-4981-af7b-ef54f58693e1-bundle\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:05.657400 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.657396 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937ea01c-5360-456b-8b4b-fd8271bc0283-util\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:05.657522 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.657403 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2738cf67-615e-4981-af7b-ef54f58693e1-util\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:05.657607 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.657585 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3eab58e-4cfa-4f2e-a440-2b998d95a384-bundle" (OuterVolumeSpecName: "bundle") pod "f3eab58e-4cfa-4f2e-a440-2b998d95a384" (UID: "f3eab58e-4cfa-4f2e-a440-2b998d95a384"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:23:05.659252 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.659230 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3eab58e-4cfa-4f2e-a440-2b998d95a384-kube-api-access-m45ng" (OuterVolumeSpecName: "kube-api-access-m45ng") pod "f3eab58e-4cfa-4f2e-a440-2b998d95a384" (UID: "f3eab58e-4cfa-4f2e-a440-2b998d95a384"). InnerVolumeSpecName "kube-api-access-m45ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:23:05.662348 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.662331 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3eab58e-4cfa-4f2e-a440-2b998d95a384-util" (OuterVolumeSpecName: "util") pod "f3eab58e-4cfa-4f2e-a440-2b998d95a384" (UID: "f3eab58e-4cfa-4f2e-a440-2b998d95a384"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:23:05.758678 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.758655 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3eab58e-4cfa-4f2e-a440-2b998d95a384-util\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:05.758678 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.758675 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m45ng\" (UniqueName: \"kubernetes.io/projected/f3eab58e-4cfa-4f2e-a440-2b998d95a384-kube-api-access-m45ng\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:05.758789 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:05.758686 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3eab58e-4cfa-4f2e-a440-2b998d95a384-bundle\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:06.293859 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.293835 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" Sep 16 01:23:06.295178 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.295136 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13bba4d5-c971-40af-b470-52b17914a115" path="/var/lib/kubelet/pods/13bba4d5-c971-40af-b470-52b17914a115/volumes" Sep 16 01:23:06.295773 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.295743 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azjg5b" event={"ID":"2738cf67-615e-4981-af7b-ef54f58693e1","Type":"ContainerDied","Data":"568729c698a4f7a52f14ab45e01ac4a6117a2f967fd17e598abf5191b19b8984"} Sep 16 01:23:06.295880 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.295789 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="568729c698a4f7a52f14ab45e01ac4a6117a2f967fd17e598abf5191b19b8984" Sep 16 01:23:06.295880 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.295797 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" Sep 16 01:23:06.295996 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.295805 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb5knx" event={"ID":"f3eab58e-4cfa-4f2e-a440-2b998d95a384","Type":"ContainerDied","Data":"c1197d872430720cf302afb07e2ae83acd4100e3288929834b3590d664bb0753"} Sep 16 01:23:06.295996 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.295933 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1197d872430720cf302afb07e2ae83acd4100e3288929834b3590d664bb0753" Sep 16 01:23:06.298323 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.298281 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" Sep 16 01:23:06.298323 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.298300 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sx5jz" event={"ID":"937ea01c-5360-456b-8b4b-fd8271bc0283","Type":"ContainerDied","Data":"5210128c19296175752de2dd7f238f7c17cc02eddea16dfca0d8e806fad06dd3"} Sep 16 01:23:06.298491 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.298345 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5210128c19296175752de2dd7f238f7c17cc02eddea16dfca0d8e806fad06dd3" Sep 16 01:23:06.433202 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.433176 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" Sep 16 01:23:06.564763 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.564681 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlhxw\" (UniqueName: \"kubernetes.io/projected/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-kube-api-access-zlhxw\") pod \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\" (UID: \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\") " Sep 16 01:23:06.564763 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.564728 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-util\") pod \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\" (UID: \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\") " Sep 16 01:23:06.564990 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.564819 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-bundle\") pod \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\" (UID: \"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3\") " Sep 16 01:23:06.565299 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.565273 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-bundle" (OuterVolumeSpecName: "bundle") pod "6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3" (UID: "6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:23:06.566778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.566748 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-kube-api-access-zlhxw" (OuterVolumeSpecName: "kube-api-access-zlhxw") pod "6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3" (UID: "6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3"). InnerVolumeSpecName "kube-api-access-zlhxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:23:06.570977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.570953 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-util" (OuterVolumeSpecName: "util") pod "6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3" (UID: "6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:23:06.665300 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.665277 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-util\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:06.665300 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.665296 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-bundle\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:06.665300 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:06.665304 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlhxw\" (UniqueName: \"kubernetes.io/projected/6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3-kube-api-access-zlhxw\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:07.305256 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:07.305223 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" event={"ID":"6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3","Type":"ContainerDied","Data":"75d6bbc8d310c1d95fb31d24105d41f3a77ef4fdcb3b240644ca495e82ed3b46"} Sep 16 01:23:07.305609 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:07.305272 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75d6bbc8d310c1d95fb31d24105d41f3a77ef4fdcb3b240644ca495e82ed3b46" Sep 16 01:23:07.305609 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:07.305233 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w2lcd" Sep 16 01:23:09.484658 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:09.484626 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:12.108067 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:12.108030 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:12.488917 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:12.488883 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:12.879322 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:12.879282 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:15.480035 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.479995 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc"] Sep 16 01:23:15.480589 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480568 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3eab58e-4cfa-4f2e-a440-2b998d95a384" containerName="util" Sep 16 01:23:15.480665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480591 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3eab58e-4cfa-4f2e-a440-2b998d95a384" containerName="util" Sep 16 01:23:15.480665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480601 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937ea01c-5360-456b-8b4b-fd8271bc0283" containerName="pull" Sep 16 01:23:15.480665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480609 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="937ea01c-5360-456b-8b4b-fd8271bc0283" containerName="pull" Sep 16 01:23:15.480665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480619 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937ea01c-5360-456b-8b4b-fd8271bc0283" containerName="util" Sep 16 01:23:15.480665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480626 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="937ea01c-5360-456b-8b4b-fd8271bc0283" containerName="util" Sep 16 01:23:15.480665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480655 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3" containerName="pull" Sep 16 01:23:15.480665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480664 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3" containerName="pull" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480684 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2738cf67-615e-4981-af7b-ef54f58693e1" containerName="util" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480693 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2738cf67-615e-4981-af7b-ef54f58693e1" containerName="util" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480703 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2738cf67-615e-4981-af7b-ef54f58693e1" containerName="extract" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480712 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2738cf67-615e-4981-af7b-ef54f58693e1" containerName="extract" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480721 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3eab58e-4cfa-4f2e-a440-2b998d95a384" containerName="pull" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480729 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3eab58e-4cfa-4f2e-a440-2b998d95a384" containerName="pull" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480740 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3" containerName="extract" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480747 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3" containerName="extract" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480758 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3" containerName="util" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480773 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3" containerName="util" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480790 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937ea01c-5360-456b-8b4b-fd8271bc0283" containerName="extract" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480798 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="937ea01c-5360-456b-8b4b-fd8271bc0283" containerName="extract" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480810 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13bba4d5-c971-40af-b470-52b17914a115" containerName="console" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480820 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="13bba4d5-c971-40af-b470-52b17914a115" containerName="console" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480831 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2738cf67-615e-4981-af7b-ef54f58693e1" containerName="pull" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480839 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2738cf67-615e-4981-af7b-ef54f58693e1" containerName="pull" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480849 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3eab58e-4cfa-4f2e-a440-2b998d95a384" containerName="extract" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480857 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3eab58e-4cfa-4f2e-a440-2b998d95a384" containerName="extract" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480947 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="13bba4d5-c971-40af-b470-52b17914a115" containerName="console" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480961 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3eab58e-4cfa-4f2e-a440-2b998d95a384" containerName="extract" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480971 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f0834b1-bb95-4ad2-8a14-f66d2dbb77d3" containerName="extract" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480982 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2738cf67-615e-4981-af7b-ef54f58693e1" containerName="extract" Sep 16 01:23:15.481030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.480991 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="937ea01c-5360-456b-8b4b-fd8271bc0283" containerName="extract" Sep 16 01:23:15.486273 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.486253 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:15.488669 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.488642 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 01:23:15.488797 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.488689 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-h9x8n\"" Sep 16 01:23:15.488797 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.488727 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 01:23:15.494569 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.494546 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc"] Sep 16 01:23:15.540960 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.540927 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd22c215-11b7-4af9-8dd5-3aa12c0852f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xjklc\" (UID: \"dd22c215-11b7-4af9-8dd5-3aa12c0852f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:15.541097 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.540982 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js4rs\" (UniqueName: \"kubernetes.io/projected/dd22c215-11b7-4af9-8dd5-3aa12c0852f9-kube-api-access-js4rs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xjklc\" (UID: \"dd22c215-11b7-4af9-8dd5-3aa12c0852f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:15.642069 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.642025 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd22c215-11b7-4af9-8dd5-3aa12c0852f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xjklc\" (UID: \"dd22c215-11b7-4af9-8dd5-3aa12c0852f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:15.642254 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.642149 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js4rs\" (UniqueName: \"kubernetes.io/projected/dd22c215-11b7-4af9-8dd5-3aa12c0852f9-kube-api-access-js4rs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xjklc\" (UID: \"dd22c215-11b7-4af9-8dd5-3aa12c0852f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:15.642507 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.642477 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd22c215-11b7-4af9-8dd5-3aa12c0852f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xjklc\" (UID: \"dd22c215-11b7-4af9-8dd5-3aa12c0852f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:15.650903 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.650881 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js4rs\" (UniqueName: \"kubernetes.io/projected/dd22c215-11b7-4af9-8dd5-3aa12c0852f9-kube-api-access-js4rs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xjklc\" (UID: \"dd22c215-11b7-4af9-8dd5-3aa12c0852f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:15.689783 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.689754 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:15.797076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.796990 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:15.927703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:15.927667 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:16.072457 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:16.072380 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:16.092651 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:16.092626 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc"] Sep 16 01:23:16.095785 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:23:16.095758 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd22c215_11b7_4af9_8dd5_3aa12c0852f9.slice/crio-44a575152e60f0f8528afe3c2f760a413094dd846dd7d44174113edb65d1f5be WatchSource:0}: Error finding container 44a575152e60f0f8528afe3c2f760a413094dd846dd7d44174113edb65d1f5be: Status 404 returned error can't find the container with id 44a575152e60f0f8528afe3c2f760a413094dd846dd7d44174113edb65d1f5be Sep 16 01:23:16.353200 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:16.353108 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" event={"ID":"dd22c215-11b7-4af9-8dd5-3aa12c0852f9","Type":"ContainerStarted","Data":"44a575152e60f0f8528afe3c2f760a413094dd846dd7d44174113edb65d1f5be"} Sep 16 01:23:16.952137 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:16.952104 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:17.056893 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:17.056864 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:18.531976 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:18.531936 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:18.703777 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:18.703709 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:19.920661 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:19.920618 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:20.300492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:20.300410 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v5wjf"] Sep 16 01:23:20.304252 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:20.304227 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v5wjf" Sep 16 01:23:20.306620 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:20.306559 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 16 01:23:20.306751 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:20.306663 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-q44lb\"" Sep 16 01:23:20.312188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:20.312146 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v5wjf"] Sep 16 01:23:20.381560 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:20.381525 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-546mx\" (UniqueName: \"kubernetes.io/projected/023e4195-ebc3-4f80-8e7b-d7ac7e145e56-kube-api-access-546mx\") pod \"dns-operator-controller-manager-7b84cd7c9c-v5wjf\" (UID: \"023e4195-ebc3-4f80-8e7b-d7ac7e145e56\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v5wjf" Sep 16 01:23:20.482620 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:20.482580 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-546mx\" (UniqueName: \"kubernetes.io/projected/023e4195-ebc3-4f80-8e7b-d7ac7e145e56-kube-api-access-546mx\") pod \"dns-operator-controller-manager-7b84cd7c9c-v5wjf\" (UID: \"023e4195-ebc3-4f80-8e7b-d7ac7e145e56\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v5wjf" Sep 16 01:23:20.494447 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:20.494421 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-546mx\" (UniqueName: \"kubernetes.io/projected/023e4195-ebc3-4f80-8e7b-d7ac7e145e56-kube-api-access-546mx\") pod \"dns-operator-controller-manager-7b84cd7c9c-v5wjf\" (UID: \"023e4195-ebc3-4f80-8e7b-d7ac7e145e56\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v5wjf" Sep 16 01:23:20.619314 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:20.619285 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v5wjf" Sep 16 01:23:21.882949 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:21.882912 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:21.983425 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:21.983395 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:22.086574 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:22.086546 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v5wjf"] Sep 16 01:23:22.088065 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:23:22.088041 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023e4195_ebc3_4f80_8e7b_d7ac7e145e56.slice/crio-4f5203d7e24c9b5b13a6b49c9d1bf8163b9f85ce1f59a38a03d6b3c6383d20cc WatchSource:0}: Error finding container 4f5203d7e24c9b5b13a6b49c9d1bf8163b9f85ce1f59a38a03d6b3c6383d20cc: Status 404 returned error can't find the container with id 4f5203d7e24c9b5b13a6b49c9d1bf8163b9f85ce1f59a38a03d6b3c6383d20cc Sep 16 01:23:22.392789 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:22.392748 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v5wjf" event={"ID":"023e4195-ebc3-4f80-8e7b-d7ac7e145e56","Type":"ContainerStarted","Data":"4f5203d7e24c9b5b13a6b49c9d1bf8163b9f85ce1f59a38a03d6b3c6383d20cc"} Sep 16 01:23:22.394383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:22.394359 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" event={"ID":"dd22c215-11b7-4af9-8dd5-3aa12c0852f9","Type":"ContainerStarted","Data":"72b249dc3f22279a3e55449a8a3e6164628ed9bffc46ead2643ff34b53e51c3f"} Sep 16 01:23:22.394520 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:22.394486 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:22.411891 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:22.411846 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" podStartSLOduration=1.6147176349999999 podStartE2EDuration="7.411832437s" totalImagesPullingTime="5.797114802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:23:22.410870471 +0000 UTC m=+432.663293060" watchObservedRunningTime="2026-09-16 01:23:22.411832437 +0000 UTC m=+432.664255027" Sep 16 01:23:24.408687 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:24.408659 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v5wjf" event={"ID":"023e4195-ebc3-4f80-8e7b-d7ac7e145e56","Type":"ContainerStarted","Data":"9c1abfd65e6ea8d18478b94fbb326f0f0a81cccf8ea32d25369c7b43142cfd4f"} Sep 16 01:23:24.409100 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:24.408779 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v5wjf" Sep 16 01:23:24.423537 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:24.423486 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v5wjf" podStartSLOduration=2.472459636 podStartE2EDuration="4.423471279s" totalImagesPullingTime="1.951011643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:23:24.422226838 +0000 UTC m=+434.674649428" watchObservedRunningTime="2026-09-16 01:23:24.423471279 +0000 UTC m=+434.675893868" Sep 16 01:23:24.754054 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:24.753982 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:25.430925 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:25.430896 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:25.682504 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:25.682382 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:25.782848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:25.782806 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:26.078915 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:26.078820 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:26.180707 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:26.180680 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:26.489092 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:26.489058 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:26.585151 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:26.585120 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:33.404147 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:33.404076 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:34.311220 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:34.311182 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:34.411685 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:34.411654 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:34.598950 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:34.598876 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77"] Sep 16 01:23:34.601618 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:34.601603 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" Sep 16 01:23:34.639477 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:34.639451 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77"] Sep 16 01:23:34.695526 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:34.695497 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j48gs\" (UniqueName: \"kubernetes.io/projected/4de2715b-0911-4c30-b6f1-17dbf095261e-kube-api-access-j48gs\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-d4h77\" (UID: \"4de2715b-0911-4c30-b6f1-17dbf095261e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" Sep 16 01:23:34.695678 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:34.695548 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4de2715b-0911-4c30-b6f1-17dbf095261e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-d4h77\" (UID: \"4de2715b-0911-4c30-b6f1-17dbf095261e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" Sep 16 01:23:34.796474 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:34.796446 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4de2715b-0911-4c30-b6f1-17dbf095261e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-d4h77\" (UID: \"4de2715b-0911-4c30-b6f1-17dbf095261e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" Sep 16 01:23:34.796634 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:34.796547 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j48gs\" (UniqueName: \"kubernetes.io/projected/4de2715b-0911-4c30-b6f1-17dbf095261e-kube-api-access-j48gs\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-d4h77\" (UID: \"4de2715b-0911-4c30-b6f1-17dbf095261e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" Sep 16 01:23:34.796907 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:34.796881 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4de2715b-0911-4c30-b6f1-17dbf095261e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-d4h77\" (UID: \"4de2715b-0911-4c30-b6f1-17dbf095261e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" Sep 16 01:23:34.805366 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:34.805344 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j48gs\" (UniqueName: \"kubernetes.io/projected/4de2715b-0911-4c30-b6f1-17dbf095261e-kube-api-access-j48gs\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-d4h77\" (UID: \"4de2715b-0911-4c30-b6f1-17dbf095261e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" Sep 16 01:23:34.906970 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:34.906943 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77"] Sep 16 01:23:34.907197 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:34.907149 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" Sep 16 01:23:35.186729 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.186560 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77"] Sep 16 01:23:35.190028 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:23:35.190001 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de2715b_0911_4c30_b6f1_17dbf095261e.slice/crio-e462a1c2f0117ea2bdeecebb9825170801eb20081489b62ed0afde5a510baec6 WatchSource:0}: Error finding container e462a1c2f0117ea2bdeecebb9825170801eb20081489b62ed0afde5a510baec6: Status 404 returned error can't find the container with id e462a1c2f0117ea2bdeecebb9825170801eb20081489b62ed0afde5a510baec6 Sep 16 01:23:35.416356 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.416327 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v5wjf" Sep 16 01:23:35.471811 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.471716 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" event={"ID":"4de2715b-0911-4c30-b6f1-17dbf095261e","Type":"ContainerStarted","Data":"ed9b11ce249660c416c042db78fedcc408c9e82e82f378673df3c8c5b6befb11"} Sep 16 01:23:35.471811 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.471779 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" event={"ID":"4de2715b-0911-4c30-b6f1-17dbf095261e","Type":"ContainerStarted","Data":"e462a1c2f0117ea2bdeecebb9825170801eb20081489b62ed0afde5a510baec6"} Sep 16 01:23:35.471811 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.471799 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" podUID="4de2715b-0911-4c30-b6f1-17dbf095261e" containerName="manager" containerID="cri-o://ed9b11ce249660c416c042db78fedcc408c9e82e82f378673df3c8c5b6befb11" gracePeriod=10 Sep 16 01:23:35.472055 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.471909 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" Sep 16 01:23:35.492320 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.492274 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" podStartSLOduration=1.492258805 podStartE2EDuration="1.492258805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:23:35.491398641 +0000 UTC m=+445.743821231" watchObservedRunningTime="2026-09-16 01:23:35.492258805 +0000 UTC m=+445.744681393" Sep 16 01:23:35.676610 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.676578 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77"] Sep 16 01:23:35.683200 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.683176 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77"] Sep 16 01:23:35.700653 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.700629 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc"] Sep 16 01:23:35.700843 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.700821 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" podUID="dd22c215-11b7-4af9-8dd5-3aa12c0852f9" containerName="manager" containerID="cri-o://72b249dc3f22279a3e55449a8a3e6164628ed9bffc46ead2643ff34b53e51c3f" gracePeriod=2 Sep 16 01:23:35.713569 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.713546 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc"] Sep 16 01:23:35.719676 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:23:35.719568 2558 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dd22c215-11b7-4af9-8dd5-3aa12c0852f9\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-xjklc\": pods \"kuadrant-operator-controller-manager-76f8f686bf-xjklc\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:35.730150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.730087 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs"] Sep 16 01:23:35.730454 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.730440 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd22c215-11b7-4af9-8dd5-3aa12c0852f9" containerName="manager" Sep 16 01:23:35.730454 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.730455 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd22c215-11b7-4af9-8dd5-3aa12c0852f9" containerName="manager" Sep 16 01:23:35.730604 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.730498 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de2715b-0911-4c30-b6f1-17dbf095261e" containerName="manager" Sep 16 01:23:35.730604 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.730514 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de2715b-0911-4c30-b6f1-17dbf095261e" containerName="manager" Sep 16 01:23:35.730719 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.730617 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd22c215-11b7-4af9-8dd5-3aa12c0852f9" containerName="manager" Sep 16 01:23:35.730719 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.730628 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4de2715b-0911-4c30-b6f1-17dbf095261e" containerName="manager" Sep 16 01:23:35.733329 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.733309 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" Sep 16 01:23:35.735302 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:23:35.735268 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xjklc\" is forbidden: User \"system:node:ip-10-0-137-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-55.ec2.internal' and this object" podUID="dd22c215-11b7-4af9-8dd5-3aa12c0852f9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:35.750060 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.750037 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs"] Sep 16 01:23:35.805021 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.804983 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8428e43-67ac-4052-bf8b-6aee9aedccb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ntwgs\" (UID: \"a8428e43-67ac-4052-bf8b-6aee9aedccb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" Sep 16 01:23:35.805204 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.805091 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv9nb\" (UniqueName: \"kubernetes.io/projected/a8428e43-67ac-4052-bf8b-6aee9aedccb8-kube-api-access-fv9nb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ntwgs\" (UID: \"a8428e43-67ac-4052-bf8b-6aee9aedccb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" Sep 16 01:23:35.906506 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.906479 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8428e43-67ac-4052-bf8b-6aee9aedccb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ntwgs\" (UID: \"a8428e43-67ac-4052-bf8b-6aee9aedccb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" Sep 16 01:23:35.906653 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.906541 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv9nb\" (UniqueName: \"kubernetes.io/projected/a8428e43-67ac-4052-bf8b-6aee9aedccb8-kube-api-access-fv9nb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ntwgs\" (UID: \"a8428e43-67ac-4052-bf8b-6aee9aedccb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" Sep 16 01:23:35.906843 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.906825 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8428e43-67ac-4052-bf8b-6aee9aedccb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ntwgs\" (UID: \"a8428e43-67ac-4052-bf8b-6aee9aedccb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" Sep 16 01:23:35.924903 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.924869 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv9nb\" (UniqueName: \"kubernetes.io/projected/a8428e43-67ac-4052-bf8b-6aee9aedccb8-kube-api-access-fv9nb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ntwgs\" (UID: \"a8428e43-67ac-4052-bf8b-6aee9aedccb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" Sep 16 01:23:35.940330 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:35.940307 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:35.942209 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:23:35.942186 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xjklc\" is forbidden: User \"system:node:ip-10-0-137-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-55.ec2.internal' and this object" podUID="dd22c215-11b7-4af9-8dd5-3aa12c0852f9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:36.007303 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.007251 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd22c215-11b7-4af9-8dd5-3aa12c0852f9-extensions-socket-volume\") pod \"dd22c215-11b7-4af9-8dd5-3aa12c0852f9\" (UID: \"dd22c215-11b7-4af9-8dd5-3aa12c0852f9\") " Sep 16 01:23:36.007303 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.007291 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js4rs\" (UniqueName: \"kubernetes.io/projected/dd22c215-11b7-4af9-8dd5-3aa12c0852f9-kube-api-access-js4rs\") pod \"dd22c215-11b7-4af9-8dd5-3aa12c0852f9\" (UID: \"dd22c215-11b7-4af9-8dd5-3aa12c0852f9\") " Sep 16 01:23:36.007777 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.007752 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd22c215-11b7-4af9-8dd5-3aa12c0852f9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dd22c215-11b7-4af9-8dd5-3aa12c0852f9" (UID: "dd22c215-11b7-4af9-8dd5-3aa12c0852f9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:23:36.009467 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.009447 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd22c215-11b7-4af9-8dd5-3aa12c0852f9-kube-api-access-js4rs" (OuterVolumeSpecName: "kube-api-access-js4rs") pod "dd22c215-11b7-4af9-8dd5-3aa12c0852f9" (UID: "dd22c215-11b7-4af9-8dd5-3aa12c0852f9"). InnerVolumeSpecName "kube-api-access-js4rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:23:36.071637 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.071613 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" Sep 16 01:23:36.108925 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.108887 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js4rs\" (UniqueName: \"kubernetes.io/projected/dd22c215-11b7-4af9-8dd5-3aa12c0852f9-kube-api-access-js4rs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:36.108925 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.108927 2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd22c215-11b7-4af9-8dd5-3aa12c0852f9-extensions-socket-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:36.294573 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.294501 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd22c215-11b7-4af9-8dd5-3aa12c0852f9" path="/var/lib/kubelet/pods/dd22c215-11b7-4af9-8dd5-3aa12c0852f9/volumes" Sep 16 01:23:36.355574 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.355550 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs"] Sep 16 01:23:36.356971 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:23:36.356949 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8428e43_67ac_4052_bf8b_6aee9aedccb8.slice/crio-146857b8f4413fc5d917c2de65fa3fca6607872f00626dc2a8a20446514fad12 WatchSource:0}: Error finding container 146857b8f4413fc5d917c2de65fa3fca6607872f00626dc2a8a20446514fad12: Status 404 returned error can't find the container with id 146857b8f4413fc5d917c2de65fa3fca6607872f00626dc2a8a20446514fad12 Sep 16 01:23:36.432725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.432705 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" Sep 16 01:23:36.478321 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.478290 2558 generic.go:356] "Generic (PLEG): container finished" podID="dd22c215-11b7-4af9-8dd5-3aa12c0852f9" containerID="72b249dc3f22279a3e55449a8a3e6164628ed9bffc46ead2643ff34b53e51c3f" exitCode=0 Sep 16 01:23:36.478455 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.478339 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjklc" Sep 16 01:23:36.478455 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.478371 2558 scope.go:117] "RemoveContainer" containerID="72b249dc3f22279a3e55449a8a3e6164628ed9bffc46ead2643ff34b53e51c3f" Sep 16 01:23:36.480550 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.480526 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" event={"ID":"a8428e43-67ac-4052-bf8b-6aee9aedccb8","Type":"ContainerStarted","Data":"4a5bed77494b97b66f50d08cfe8b1985f5cb99308621b717b2791a2d5eea18e4"} Sep 16 01:23:36.480709 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.480689 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" Sep 16 01:23:36.480806 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.480735 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" event={"ID":"a8428e43-67ac-4052-bf8b-6aee9aedccb8","Type":"ContainerStarted","Data":"146857b8f4413fc5d917c2de65fa3fca6607872f00626dc2a8a20446514fad12"} Sep 16 01:23:36.482078 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.482057 2558 generic.go:356] "Generic (PLEG): container finished" podID="4de2715b-0911-4c30-b6f1-17dbf095261e" containerID="ed9b11ce249660c416c042db78fedcc408c9e82e82f378673df3c8c5b6befb11" exitCode=0 Sep 16 01:23:36.482209 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.482107 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" Sep 16 01:23:36.488123 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.488108 2558 scope.go:117] "RemoveContainer" containerID="72b249dc3f22279a3e55449a8a3e6164628ed9bffc46ead2643ff34b53e51c3f" Sep 16 01:23:36.488433 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:23:36.488413 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b249dc3f22279a3e55449a8a3e6164628ed9bffc46ead2643ff34b53e51c3f\": container with ID starting with 72b249dc3f22279a3e55449a8a3e6164628ed9bffc46ead2643ff34b53e51c3f not found: ID does not exist" containerID="72b249dc3f22279a3e55449a8a3e6164628ed9bffc46ead2643ff34b53e51c3f" Sep 16 01:23:36.488510 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.488443 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b249dc3f22279a3e55449a8a3e6164628ed9bffc46ead2643ff34b53e51c3f"} err="failed to get container status \"72b249dc3f22279a3e55449a8a3e6164628ed9bffc46ead2643ff34b53e51c3f\": rpc error: code = NotFound desc = could not find container \"72b249dc3f22279a3e55449a8a3e6164628ed9bffc46ead2643ff34b53e51c3f\": container with ID starting with 72b249dc3f22279a3e55449a8a3e6164628ed9bffc46ead2643ff34b53e51c3f not found: ID does not exist" Sep 16 01:23:36.488510 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.488462 2558 scope.go:117] "RemoveContainer" containerID="ed9b11ce249660c416c042db78fedcc408c9e82e82f378673df3c8c5b6befb11" Sep 16 01:23:36.496504 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.496486 2558 scope.go:117] "RemoveContainer" containerID="ed9b11ce249660c416c042db78fedcc408c9e82e82f378673df3c8c5b6befb11" Sep 16 01:23:36.496748 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:23:36.496731 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed9b11ce249660c416c042db78fedcc408c9e82e82f378673df3c8c5b6befb11\": container with ID starting with ed9b11ce249660c416c042db78fedcc408c9e82e82f378673df3c8c5b6befb11 not found: ID does not exist" containerID="ed9b11ce249660c416c042db78fedcc408c9e82e82f378673df3c8c5b6befb11" Sep 16 01:23:36.496805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.496754 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed9b11ce249660c416c042db78fedcc408c9e82e82f378673df3c8c5b6befb11"} err="failed to get container status \"ed9b11ce249660c416c042db78fedcc408c9e82e82f378673df3c8c5b6befb11\": rpc error: code = NotFound desc = could not find container \"ed9b11ce249660c416c042db78fedcc408c9e82e82f378673df3c8c5b6befb11\": container with ID starting with ed9b11ce249660c416c042db78fedcc408c9e82e82f378673df3c8c5b6befb11 not found: ID does not exist" Sep 16 01:23:36.505473 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.505439 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" podStartSLOduration=1.505427111 podStartE2EDuration="1.505427111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:23:36.503546562 +0000 UTC m=+446.755969153" watchObservedRunningTime="2026-09-16 01:23:36.505427111 +0000 UTC m=+446.757849699" Sep 16 01:23:36.511200 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.511183 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4de2715b-0911-4c30-b6f1-17dbf095261e-extensions-socket-volume\") pod \"4de2715b-0911-4c30-b6f1-17dbf095261e\" (UID: \"4de2715b-0911-4c30-b6f1-17dbf095261e\") " Sep 16 01:23:36.511292 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.511269 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j48gs\" (UniqueName: \"kubernetes.io/projected/4de2715b-0911-4c30-b6f1-17dbf095261e-kube-api-access-j48gs\") pod \"4de2715b-0911-4c30-b6f1-17dbf095261e\" (UID: \"4de2715b-0911-4c30-b6f1-17dbf095261e\") " Sep 16 01:23:36.511540 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.511519 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de2715b-0911-4c30-b6f1-17dbf095261e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4de2715b-0911-4c30-b6f1-17dbf095261e" (UID: "4de2715b-0911-4c30-b6f1-17dbf095261e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:23:36.513516 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.513450 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de2715b-0911-4c30-b6f1-17dbf095261e-kube-api-access-j48gs" (OuterVolumeSpecName: "kube-api-access-j48gs") pod "4de2715b-0911-4c30-b6f1-17dbf095261e" (UID: "4de2715b-0911-4c30-b6f1-17dbf095261e"). InnerVolumeSpecName "kube-api-access-j48gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:23:36.612880 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.612796 2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4de2715b-0911-4c30-b6f1-17dbf095261e-extensions-socket-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:36.612880 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:36.612834 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j48gs\" (UniqueName: \"kubernetes.io/projected/4de2715b-0911-4c30-b6f1-17dbf095261e-kube-api-access-j48gs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:36.792712 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:23:36.792680 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-d4h77\" is forbidden: User \"system:node:ip-10-0-137-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-55.ec2.internal' and this object" podUID="4de2715b-0911-4c30-b6f1-17dbf095261e" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-d4h77" Sep 16 01:23:37.634068 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:37.634038 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:37.723989 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:37.723961 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:38.294786 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:38.294757 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4de2715b-0911-4c30-b6f1-17dbf095261e" path="/var/lib/kubelet/pods/4de2715b-0911-4c30-b6f1-17dbf095261e/volumes" Sep 16 01:23:43.570571 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:43.570533 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:43.671277 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:43.671247 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:47.490691 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:47.490656 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" Sep 16 01:23:48.176471 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:48.176443 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:48.276532 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:48.276506 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:48.771757 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:48.771722 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:48.872585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:48.872554 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:49.380206 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:49.380151 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:49.481535 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:49.481509 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:52.123484 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.123448 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs"] Sep 16 01:23:52.123897 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.123661 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" podUID="a8428e43-67ac-4052-bf8b-6aee9aedccb8" containerName="manager" containerID="cri-o://4a5bed77494b97b66f50d08cfe8b1985f5cb99308621b717b2791a2d5eea18e4" gracePeriod=10 Sep 16 01:23:52.363804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.363772 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24"] Sep 16 01:23:52.367315 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.367297 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" Sep 16 01:23:52.377845 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.377735 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" Sep 16 01:23:52.381079 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.381035 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24"] Sep 16 01:23:52.439397 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.439368 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2c98573-5b4a-4907-a4e8-9a13cbf37723-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sfj24\" (UID: \"d2c98573-5b4a-4907-a4e8-9a13cbf37723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" Sep 16 01:23:52.439577 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.439419 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wf7f\" (UniqueName: \"kubernetes.io/projected/d2c98573-5b4a-4907-a4e8-9a13cbf37723-kube-api-access-9wf7f\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sfj24\" (UID: \"d2c98573-5b4a-4907-a4e8-9a13cbf37723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" Sep 16 01:23:52.539703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.539675 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv9nb\" (UniqueName: \"kubernetes.io/projected/a8428e43-67ac-4052-bf8b-6aee9aedccb8-kube-api-access-fv9nb\") pod \"a8428e43-67ac-4052-bf8b-6aee9aedccb8\" (UID: \"a8428e43-67ac-4052-bf8b-6aee9aedccb8\") " Sep 16 01:23:52.539825 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.539729 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8428e43-67ac-4052-bf8b-6aee9aedccb8-extensions-socket-volume\") pod \"a8428e43-67ac-4052-bf8b-6aee9aedccb8\" (UID: \"a8428e43-67ac-4052-bf8b-6aee9aedccb8\") " Sep 16 01:23:52.539875 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.539835 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2c98573-5b4a-4907-a4e8-9a13cbf37723-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sfj24\" (UID: \"d2c98573-5b4a-4907-a4e8-9a13cbf37723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" Sep 16 01:23:52.539929 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.539872 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wf7f\" (UniqueName: \"kubernetes.io/projected/d2c98573-5b4a-4907-a4e8-9a13cbf37723-kube-api-access-9wf7f\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sfj24\" (UID: \"d2c98573-5b4a-4907-a4e8-9a13cbf37723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" Sep 16 01:23:52.540150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.540123 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8428e43-67ac-4052-bf8b-6aee9aedccb8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a8428e43-67ac-4052-bf8b-6aee9aedccb8" (UID: "a8428e43-67ac-4052-bf8b-6aee9aedccb8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:23:52.540274 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.540249 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2c98573-5b4a-4907-a4e8-9a13cbf37723-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sfj24\" (UID: \"d2c98573-5b4a-4907-a4e8-9a13cbf37723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" Sep 16 01:23:52.541928 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.541909 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8428e43-67ac-4052-bf8b-6aee9aedccb8-kube-api-access-fv9nb" (OuterVolumeSpecName: "kube-api-access-fv9nb") pod "a8428e43-67ac-4052-bf8b-6aee9aedccb8" (UID: "a8428e43-67ac-4052-bf8b-6aee9aedccb8"). InnerVolumeSpecName "kube-api-access-fv9nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:23:52.547069 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.547051 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wf7f\" (UniqueName: \"kubernetes.io/projected/d2c98573-5b4a-4907-a4e8-9a13cbf37723-kube-api-access-9wf7f\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sfj24\" (UID: \"d2c98573-5b4a-4907-a4e8-9a13cbf37723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" Sep 16 01:23:52.571594 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.571563 2558 generic.go:356] "Generic (PLEG): container finished" podID="a8428e43-67ac-4052-bf8b-6aee9aedccb8" containerID="4a5bed77494b97b66f50d08cfe8b1985f5cb99308621b717b2791a2d5eea18e4" exitCode=0 Sep 16 01:23:52.571697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.571610 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" Sep 16 01:23:52.571697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.571644 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" event={"ID":"a8428e43-67ac-4052-bf8b-6aee9aedccb8","Type":"ContainerDied","Data":"4a5bed77494b97b66f50d08cfe8b1985f5cb99308621b717b2791a2d5eea18e4"} Sep 16 01:23:52.571808 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.571709 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs" event={"ID":"a8428e43-67ac-4052-bf8b-6aee9aedccb8","Type":"ContainerDied","Data":"146857b8f4413fc5d917c2de65fa3fca6607872f00626dc2a8a20446514fad12"} Sep 16 01:23:52.571808 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.571741 2558 scope.go:117] "RemoveContainer" containerID="4a5bed77494b97b66f50d08cfe8b1985f5cb99308621b717b2791a2d5eea18e4" Sep 16 01:23:52.581112 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.581097 2558 scope.go:117] "RemoveContainer" containerID="4a5bed77494b97b66f50d08cfe8b1985f5cb99308621b717b2791a2d5eea18e4" Sep 16 01:23:52.581405 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:23:52.581387 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5bed77494b97b66f50d08cfe8b1985f5cb99308621b717b2791a2d5eea18e4\": container with ID starting with 4a5bed77494b97b66f50d08cfe8b1985f5cb99308621b717b2791a2d5eea18e4 not found: ID does not exist" containerID="4a5bed77494b97b66f50d08cfe8b1985f5cb99308621b717b2791a2d5eea18e4" Sep 16 01:23:52.581457 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.581413 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5bed77494b97b66f50d08cfe8b1985f5cb99308621b717b2791a2d5eea18e4"} err="failed to get container status \"4a5bed77494b97b66f50d08cfe8b1985f5cb99308621b717b2791a2d5eea18e4\": rpc error: code = NotFound desc = could not find container \"4a5bed77494b97b66f50d08cfe8b1985f5cb99308621b717b2791a2d5eea18e4\": container with ID starting with 4a5bed77494b97b66f50d08cfe8b1985f5cb99308621b717b2791a2d5eea18e4 not found: ID does not exist" Sep 16 01:23:52.594503 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.594479 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs"] Sep 16 01:23:52.598393 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.598369 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ntwgs"] Sep 16 01:23:52.640607 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.640557 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv9nb\" (UniqueName: \"kubernetes.io/projected/a8428e43-67ac-4052-bf8b-6aee9aedccb8-kube-api-access-fv9nb\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:52.640607 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.640578 2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8428e43-67ac-4052-bf8b-6aee9aedccb8-extensions-socket-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:23:52.683605 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.683583 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" Sep 16 01:23:52.968275 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:52.968207 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24"] Sep 16 01:23:52.971709 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:23:52.971685 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c98573_5b4a_4907_a4e8_9a13cbf37723.slice/crio-09f01786627f86f165b834fae99d24855aa658e896474b9fdfbcf86cba43a377 WatchSource:0}: Error finding container 09f01786627f86f165b834fae99d24855aa658e896474b9fdfbcf86cba43a377: Status 404 returned error can't find the container with id 09f01786627f86f165b834fae99d24855aa658e896474b9fdfbcf86cba43a377 Sep 16 01:23:53.581234 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:53.581195 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" event={"ID":"d2c98573-5b4a-4907-a4e8-9a13cbf37723","Type":"ContainerStarted","Data":"69d6d4b975f1653b44735d76a384a00536e4ac300ce4519d76b33c40364b3d43"} Sep 16 01:23:53.581688 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:53.581259 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" event={"ID":"d2c98573-5b4a-4907-a4e8-9a13cbf37723","Type":"ContainerStarted","Data":"09f01786627f86f165b834fae99d24855aa658e896474b9fdfbcf86cba43a377"} Sep 16 01:23:53.581688 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:53.581289 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" Sep 16 01:23:53.619690 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:53.619643 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" podStartSLOduration=1.61962823 podStartE2EDuration="1.61962823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:23:53.616283456 +0000 UTC m=+463.868706045" watchObservedRunningTime="2026-09-16 01:23:53.61962823 +0000 UTC m=+463.872050818" Sep 16 01:23:54.293974 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:54.293940 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8428e43-67ac-4052-bf8b-6aee9aedccb8" path="/var/lib/kubelet/pods/a8428e43-67ac-4052-bf8b-6aee9aedccb8/volumes" Sep 16 01:23:57.583342 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:57.583303 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:57.641348 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:57.641321 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:57.773002 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:57.772973 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:23:57.884680 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:23:57.884654 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:00.570455 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:00.570418 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:00.623105 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:00.623073 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:04.589658 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:04.589626 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" Sep 16 01:24:08.544379 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.544347 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c"] Sep 16 01:24:08.544822 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.544805 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8428e43-67ac-4052-bf8b-6aee9aedccb8" containerName="manager" Sep 16 01:24:08.544871 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.544825 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8428e43-67ac-4052-bf8b-6aee9aedccb8" containerName="manager" Sep 16 01:24:08.544903 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.544893 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8428e43-67ac-4052-bf8b-6aee9aedccb8" containerName="manager" Sep 16 01:24:08.547870 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.547853 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.550077 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.550056 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hz2lg\"" Sep 16 01:24:08.550077 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.550071 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 01:24:08.550296 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.550225 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 01:24:08.550296 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.550249 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 01:24:08.559010 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.558988 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c"] Sep 16 01:24:08.566380 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.566356 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:08.663578 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.663543 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.663736 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.663584 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.663736 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.663615 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.663736 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.663661 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.663736 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.663681 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99r6\" (UniqueName: \"kubernetes.io/projected/cf21b222-a313-4498-8f0c-016d7fe3656a-kube-api-access-q99r6\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.663736 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.663709 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf21b222-a313-4498-8f0c-016d7fe3656a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.663926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.663750 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf21b222-a313-4498-8f0c-016d7fe3656a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.663926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.663765 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf21b222-a313-4498-8f0c-016d7fe3656a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.663926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.663817 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.764579 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.764549 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.764728 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.764585 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.764728 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.764606 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.764728 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.764628 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.764728 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.764645 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99r6\" (UniqueName: \"kubernetes.io/projected/cf21b222-a313-4498-8f0c-016d7fe3656a-kube-api-access-q99r6\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.764728 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.764672 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf21b222-a313-4498-8f0c-016d7fe3656a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.764728 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.764724 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf21b222-a313-4498-8f0c-016d7fe3656a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.765058 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.764746 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf21b222-a313-4498-8f0c-016d7fe3656a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.765058 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.764800 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.765058 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.764983 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.765058 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.765036 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.765289 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.765148 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.765289 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.765260 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.765560 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.765476 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf21b222-a313-4498-8f0c-016d7fe3656a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.767031 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.767006 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf21b222-a313-4498-8f0c-016d7fe3656a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.767271 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.767253 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf21b222-a313-4498-8f0c-016d7fe3656a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.771943 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.771918 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf21b222-a313-4498-8f0c-016d7fe3656a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.772134 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.772118 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99r6\" (UniqueName: \"kubernetes.io/projected/cf21b222-a313-4498-8f0c-016d7fe3656a-kube-api-access-q99r6\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7tf4c\" (UID: \"cf21b222-a313-4498-8f0c-016d7fe3656a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:08.865917 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:08.865895 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:09.140981 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:09.140955 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c"] Sep 16 01:24:09.142731 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:24:09.142703 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf21b222_a313_4498_8f0c_016d7fe3656a.slice/crio-a00522d571dd346bd19ba2a99507d83f1ccc98d283ee7f3fde94bcd12b5a079c WatchSource:0}: Error finding container a00522d571dd346bd19ba2a99507d83f1ccc98d283ee7f3fde94bcd12b5a079c: Status 404 returned error can't find the container with id a00522d571dd346bd19ba2a99507d83f1ccc98d283ee7f3fde94bcd12b5a079c Sep 16 01:24:09.144888 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:09.144851 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:24:09.144957 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:09.144917 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:24:09.144957 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:09.144946 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:24:09.496501 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:09.496424 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:09.668591 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:09.668555 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" event={"ID":"cf21b222-a313-4498-8f0c-016d7fe3656a","Type":"ContainerStarted","Data":"09598a94a56a236cd4924ff75547be6b47f507e4742297dd1cd133808837777e"} Sep 16 01:24:09.668954 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:09.668608 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" event={"ID":"cf21b222-a313-4498-8f0c-016d7fe3656a","Type":"ContainerStarted","Data":"a00522d571dd346bd19ba2a99507d83f1ccc98d283ee7f3fde94bcd12b5a079c"} Sep 16 01:24:09.689420 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:09.689319 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" podStartSLOduration=1.6893009509999999 podStartE2EDuration="1.689300951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:24:09.688732343 +0000 UTC m=+479.941154933" watchObservedRunningTime="2026-09-16 01:24:09.689300951 +0000 UTC m=+479.941723541" Sep 16 01:24:09.866091 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:09.866060 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:10.871755 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:10.871730 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:11.678592 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:11.678561 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:11.679449 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:11.679430 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:24:13.946548 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:13.946517 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:13.999275 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:13.999246 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:26.611243 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:26.611207 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:26.713649 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:26.713609 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:26.811248 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:26.811205 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:26.921100 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:26.921067 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:24:26.925778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:26.925760 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-btqdv" Sep 16 01:24:26.927928 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:26.927907 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 16 01:24:26.928027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:26.928007 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-q65k2\"" Sep 16 01:24:26.931555 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:26.931532 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:24:26.958470 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:26.958447 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:24:27.098670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:27.098634 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rgns\" (UniqueName: \"kubernetes.io/projected/5b2e403b-f3b8-46a9-b893-40a74db00bf8-kube-api-access-4rgns\") pod \"limitador-limitador-84994f7ffc-btqdv\" (UID: \"5b2e403b-f3b8-46a9-b893-40a74db00bf8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-btqdv" Sep 16 01:24:27.098825 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:27.098748 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b2e403b-f3b8-46a9-b893-40a74db00bf8-config-file\") pod \"limitador-limitador-84994f7ffc-btqdv\" (UID: \"5b2e403b-f3b8-46a9-b893-40a74db00bf8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-btqdv" Sep 16 01:24:27.199748 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:27.199686 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rgns\" (UniqueName: \"kubernetes.io/projected/5b2e403b-f3b8-46a9-b893-40a74db00bf8-kube-api-access-4rgns\") pod \"limitador-limitador-84994f7ffc-btqdv\" (UID: \"5b2e403b-f3b8-46a9-b893-40a74db00bf8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-btqdv" Sep 16 01:24:27.199748 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:27.199743 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b2e403b-f3b8-46a9-b893-40a74db00bf8-config-file\") pod \"limitador-limitador-84994f7ffc-btqdv\" (UID: \"5b2e403b-f3b8-46a9-b893-40a74db00bf8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-btqdv" Sep 16 01:24:27.200347 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:27.200329 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b2e403b-f3b8-46a9-b893-40a74db00bf8-config-file\") pod \"limitador-limitador-84994f7ffc-btqdv\" (UID: \"5b2e403b-f3b8-46a9-b893-40a74db00bf8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-btqdv" Sep 16 01:24:27.207717 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:27.207692 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rgns\" (UniqueName: \"kubernetes.io/projected/5b2e403b-f3b8-46a9-b893-40a74db00bf8-kube-api-access-4rgns\") pod \"limitador-limitador-84994f7ffc-btqdv\" (UID: \"5b2e403b-f3b8-46a9-b893-40a74db00bf8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-btqdv" Sep 16 01:24:27.237426 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:27.237402 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-btqdv" Sep 16 01:24:27.408205 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:27.408173 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:27.497200 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:27.497117 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:27.506034 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:27.506009 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:24:27.507937 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:24:27.507913 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2e403b_f3b8_46a9_b893_40a74db00bf8.slice/crio-fcd161802630ce34cbcfefc61de370d19ae88c5975427434b775d714bc0f50c1 WatchSource:0}: Error finding container fcd161802630ce34cbcfefc61de370d19ae88c5975427434b775d714bc0f50c1: Status 404 returned error can't find the container with id fcd161802630ce34cbcfefc61de370d19ae88c5975427434b775d714bc0f50c1 Sep 16 01:24:27.769424 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:27.769341 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-btqdv" event={"ID":"5b2e403b-f3b8-46a9-b893-40a74db00bf8","Type":"ContainerStarted","Data":"fcd161802630ce34cbcfefc61de370d19ae88c5975427434b775d714bc0f50c1"} Sep 16 01:24:28.776931 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:28.776857 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-btqdv" event={"ID":"5b2e403b-f3b8-46a9-b893-40a74db00bf8","Type":"ContainerStarted","Data":"94006051b38f530b14eeb0968b9ea164e7d266b789416ccee595c0e4ab334315"} Sep 16 01:24:28.777308 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:28.777041 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-btqdv" Sep 16 01:24:28.792180 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:28.792117 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-btqdv" podStartSLOduration=1.8516407049999999 podStartE2EDuration="2.792103081s" totalImagesPullingTime="940.462376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:24:28.791334797 +0000 UTC m=+499.043757399" watchObservedRunningTime="2026-09-16 01:24:28.792103081 +0000 UTC m=+499.044525672" Sep 16 01:24:39.784029 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:39.784001 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-btqdv" Sep 16 01:24:42.247241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:42.247204 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:42.346192 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:42.346147 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:42.748400 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:42.748366 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:52.711482 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:52.711449 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:24:53.027176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:24:53.027078 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:00.175991 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:00.175958 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:00.231563 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:00.231536 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:00.280502 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:00.280467 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:00.410589 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:00.410543 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:00.754843 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:00.754817 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:04.254760 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:04.254676 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:04.355089 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:04.355062 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:15.636283 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:15.636251 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:17.284611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:17.284578 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:35.843648 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:35.843607 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:35.942495 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:35.942464 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:37.278367 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:37.278334 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:37.379292 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:37.379262 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:37.694450 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:37.694415 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:37.796309 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:37.796275 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:37.899064 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:37.899029 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:38.049640 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:38.049550 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:38.141387 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:38.141354 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:25:47.106439 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:47.106408 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:48.225988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:48.225956 2558 generic.go:356] "Generic (PLEG): container finished" podID="cf21b222-a313-4498-8f0c-016d7fe3656a" containerID="09598a94a56a236cd4924ff75547be6b47f507e4742297dd1cd133808837777e" exitCode=0 Sep 16 01:25:48.226381 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:48.226028 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" event={"ID":"cf21b222-a313-4498-8f0c-016d7fe3656a","Type":"ContainerDied","Data":"09598a94a56a236cd4924ff75547be6b47f507e4742297dd1cd133808837777e"} Sep 16 01:25:48.226429 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:48.226392 2558 scope.go:117] "RemoveContainer" containerID="09598a94a56a236cd4924ff75547be6b47f507e4742297dd1cd133808837777e" Sep 16 01:25:48.227424 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:48.227374 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:25:48.227479 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:48.227463 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:25:48.227524 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:48.227494 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:25:48.676487 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:48.676446 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:48.776501 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:48.776458 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:25:49.233332 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:49.233294 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" event={"ID":"cf21b222-a313-4498-8f0c-016d7fe3656a","Type":"ContainerStarted","Data":"9512971550e3b5b9fba4f1280eeb0e8f8fd2a459fadfa2485ea9309846554a14"} Sep 16 01:25:49.866232 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:49.866086 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:25:49.866873 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:49.866532 2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7tf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 16 01:25:49.866873 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:49.866592 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" podUID="cf21b222-a313-4498-8f0c-016d7fe3656a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 16 01:25:50.866569 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:50.866532 2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7tf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 16 01:25:50.867068 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:50.866599 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" podUID="cf21b222-a313-4498-8f0c-016d7fe3656a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 16 01:25:51.867292 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:51.867253 2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7tf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 16 01:25:51.867790 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:51.867340 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" podUID="cf21b222-a313-4498-8f0c-016d7fe3656a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 16 01:25:52.867302 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:52.867260 2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7tf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 16 01:25:52.867872 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:52.867837 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" podUID="cf21b222-a313-4498-8f0c-016d7fe3656a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 16 01:25:53.867292 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:53.867257 2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7tf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 16 01:25:53.867582 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:53.867553 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" podUID="cf21b222-a313-4498-8f0c-016d7fe3656a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 16 01:25:54.870594 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:54.870565 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:25:54.870958 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:54.870850 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:25:54.871611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:25:54.871595 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7tf4c" Sep 16 01:26:01.414909 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:01.414871 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:26:01.718413 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:01.718335 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:26:10.206674 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:10.206647 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:26:10.207253 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:10.207231 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:26:22.449215 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:22.449178 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:26:23.217036 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:23.217000 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:26:29.347794 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:29.347757 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:26:30.216934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:30.216901 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:26:37.128701 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:37.128620 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:26:38.313919 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:38.313887 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:26:45.012869 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.012836 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq"] Sep 16 01:26:45.015428 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.015403 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.018241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.018216 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 16 01:26:45.018380 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.018245 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 16 01:26:45.018380 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.018274 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lbkhj\"" Sep 16 01:26:45.018380 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.018245 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 16 01:26:45.025538 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.025514 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq"] Sep 16 01:26:45.101053 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.101012 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qjxg\" (UniqueName: \"kubernetes.io/projected/9cb542bb-4011-402e-be93-80cd27255f7a-kube-api-access-7qjxg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.101234 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.101067 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cb542bb-4011-402e-be93-80cd27255f7a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.101234 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.101084 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cb542bb-4011-402e-be93-80cd27255f7a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.101234 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.101148 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cb542bb-4011-402e-be93-80cd27255f7a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.101370 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.101254 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cb542bb-4011-402e-be93-80cd27255f7a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.101370 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.101273 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cb542bb-4011-402e-be93-80cd27255f7a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.120597 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.120569 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:26:45.202268 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.202233 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cb542bb-4011-402e-be93-80cd27255f7a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.202268 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.202270 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cb542bb-4011-402e-be93-80cd27255f7a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.202482 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.202326 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qjxg\" (UniqueName: \"kubernetes.io/projected/9cb542bb-4011-402e-be93-80cd27255f7a-kube-api-access-7qjxg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.202482 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.202374 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cb542bb-4011-402e-be93-80cd27255f7a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.202482 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.202396 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cb542bb-4011-402e-be93-80cd27255f7a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.202482 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.202419 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cb542bb-4011-402e-be93-80cd27255f7a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.202715 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.202692 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cb542bb-4011-402e-be93-80cd27255f7a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.202752 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.202723 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cb542bb-4011-402e-be93-80cd27255f7a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.202786 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.202760 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cb542bb-4011-402e-be93-80cd27255f7a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.204803 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.204776 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cb542bb-4011-402e-be93-80cd27255f7a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.205132 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.205113 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cb542bb-4011-402e-be93-80cd27255f7a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.209422 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.209399 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qjxg\" (UniqueName: \"kubernetes.io/projected/9cb542bb-4011-402e-be93-80cd27255f7a-kube-api-access-7qjxg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq\" (UID: \"9cb542bb-4011-402e-be93-80cd27255f7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.327715 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.327644 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:45.600386 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.600365 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq"] Sep 16 01:26:45.602642 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:26:45.602614 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb542bb_4011_402e_be93_80cd27255f7a.slice/crio-14f801df7757c53f82d66b95a0cc83719cf4065884cbbf77698fc83f656b2c1b WatchSource:0}: Error finding container 14f801df7757c53f82d66b95a0cc83719cf4065884cbbf77698fc83f656b2c1b: Status 404 returned error can't find the container with id 14f801df7757c53f82d66b95a0cc83719cf4065884cbbf77698fc83f656b2c1b Sep 16 01:26:45.604002 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:45.603982 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 01:26:46.557948 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:46.557917 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" event={"ID":"9cb542bb-4011-402e-be93-80cd27255f7a","Type":"ContainerStarted","Data":"14f801df7757c53f82d66b95a0cc83719cf4065884cbbf77698fc83f656b2c1b"} Sep 16 01:26:46.816643 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:46.816554 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:26:47.567086 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:47.567045 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" event={"ID":"9cb542bb-4011-402e-be93-80cd27255f7a","Type":"ContainerStarted","Data":"13eb5aeb33d0c3974c7439b5a4d6c49f6251a80c4f5100000afebbe6997ee496"} Sep 16 01:26:47.583329 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:47.583288 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" podStartSLOduration=2.617461194 podStartE2EDuration="3.583271905s" totalImagesPullingTime="965.810711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:26:47.582531745 +0000 UTC m=+637.834954347" watchObservedRunningTime="2026-09-16 01:26:47.583271905 +0000 UTC m=+637.835694494" Sep 16 01:26:55.328571 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:55.328541 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:55.328987 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:55.328594 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:55.341112 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:55.341089 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:26:55.621081 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:26:55.621058 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsfdcq" Sep 16 01:27:00.625848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:00.625814 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:27:02.222501 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:02.222472 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:27:10.517327 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.517298 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr"] Sep 16 01:27:10.519687 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.519672 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.521832 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.521812 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 16 01:27:10.528613 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.528588 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr"] Sep 16 01:27:10.607507 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.607401 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.608057 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.608035 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.608288 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.608270 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl97c\" (UniqueName: \"kubernetes.io/projected/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-kube-api-access-hl97c\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.608636 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.608571 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.608884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.608865 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.609021 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.609006 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.610820 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.610795 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:27:10.709759 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.709732 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.709936 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.709768 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.709936 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.709789 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.709936 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.709810 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.709936 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.709840 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.709936 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.709876 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl97c\" (UniqueName: \"kubernetes.io/projected/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-kube-api-access-hl97c\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.710264 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.710239 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.710371 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.710342 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.710502 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.710410 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.712324 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.712299 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.712478 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.712457 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.717233 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.717206 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl97c\" (UniqueName: \"kubernetes.io/projected/1595cbbd-1960-4e6c-9a6a-c2b1fe452421-kube-api-access-hl97c\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr\" (UID: \"1595cbbd-1960-4e6c-9a6a-c2b1fe452421\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:10.829939 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:10.829870 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:11.002027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:11.001997 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:27:11.102206 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:11.102170 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr"] Sep 16 01:27:11.103611 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:27:11.103584 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1595cbbd_1960_4e6c_9a6a_c2b1fe452421.slice/crio-cf72165ce9c689be50b28347039219a704a2931bd1ea10cf324b83c16e777d0c WatchSource:0}: Error finding container cf72165ce9c689be50b28347039219a704a2931bd1ea10cf324b83c16e777d0c: Status 404 returned error can't find the container with id cf72165ce9c689be50b28347039219a704a2931bd1ea10cf324b83c16e777d0c Sep 16 01:27:11.106683 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:11.106657 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:27:11.698015 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:11.697981 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" event={"ID":"1595cbbd-1960-4e6c-9a6a-c2b1fe452421","Type":"ContainerStarted","Data":"05adf15edd5c58c4ebd912d9b420cc15ac003098a999f62dcda252167fb5476b"} Sep 16 01:27:11.698015 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:11.698021 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" event={"ID":"1595cbbd-1960-4e6c-9a6a-c2b1fe452421","Type":"ContainerStarted","Data":"cf72165ce9c689be50b28347039219a704a2931bd1ea10cf324b83c16e777d0c"} Sep 16 01:27:11.714966 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:11.714922 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" podStartSLOduration=1.549385414 podStartE2EDuration="1.714906514s" totalImagesPullingTime="165.5211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:27:11.713254832 +0000 UTC m=+661.965677422" watchObservedRunningTime="2026-09-16 01:27:11.714906514 +0000 UTC m=+661.967329103" Sep 16 01:27:12.816998 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:12.816964 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:27:20.830225 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:20.830191 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:20.830225 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:20.830233 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:20.842442 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:20.842417 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:27:21.764199 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:27:21.764152 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pmqr" Sep 16 01:28:10.779379 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:28:10.779303 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:28:11.238413 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:28:11.238380 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:28:11.382251 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:28:11.382222 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:28:27.911713 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:28:27.911679 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:28:28.001421 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:28:28.001371 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:28:53.746577 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:28:53.746548 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:28:54.101917 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:28:54.101821 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:28:55.050789 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:28:55.050758 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:28:55.151923 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:28:55.151890 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:28:56.215186 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:28:56.215136 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:28:56.314096 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:28:56.314063 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:15.861559 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:15.861525 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:15.912731 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:15.912704 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:16.189593 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.189559 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:16.209174 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.209124 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7"] Sep 16 01:29:16.212681 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.212645 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.215270 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.215244 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-482177-openshift-default-dockercfg-lcm8x\"" Sep 16 01:29:16.221602 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.221578 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7"] Sep 16 01:29:16.269015 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.268981 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:16.307506 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.307476 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-data\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.307506 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.307506 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-token\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.307738 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.307532 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-envoy\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.307738 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.307562 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9h7j\" (UniqueName: \"kubernetes.io/projected/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-kube-api-access-x9h7j\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.307738 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.307607 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-credential-socket\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.307738 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.307625 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-workload-socket\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.307738 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.307691 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-workload-certs\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.307738 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.307718 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-podinfo\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.307738 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.307733 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istiod-ca-cert\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.392206 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.392142 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:16.408571 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.408540 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-credential-socket\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.408571 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.408576 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-workload-socket\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.408802 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.408621 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-workload-certs\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.408802 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.408637 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-podinfo\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.408802 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.408653 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istiod-ca-cert\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.408802 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.408693 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-data\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.408802 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.408715 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-token\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.408802 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.408752 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-envoy\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.408802 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.408785 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9h7j\" (UniqueName: \"kubernetes.io/projected/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-kube-api-access-x9h7j\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.409205 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.408997 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-credential-socket\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.409205 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.409064 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-workload-certs\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.409205 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.409098 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-data\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.409343 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.409203 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-workload-socket\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.409436 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.409419 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istiod-ca-cert\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.411266 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.411242 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-envoy\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.411403 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.411383 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-podinfo\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.422664 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.422643 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-token\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.422793 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.422740 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9h7j\" (UniqueName: \"kubernetes.io/projected/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-kube-api-access-x9h7j\") pod \"e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.490685 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.490590 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:16.525788 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.525760 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:16.699510 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.699464 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:16.810427 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.810403 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7"] Sep 16 01:29:16.812299 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:29:16.812274 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabbd6e5_e0cc_4b5d_b1a0_e1180476a99b.slice/crio-bb8a90c737f96aa458830e50b75b01a738cdadc216c69a937f94ba8519b46e8f WatchSource:0}: Error finding container bb8a90c737f96aa458830e50b75b01a738cdadc216c69a937f94ba8519b46e8f: Status 404 returned error can't find the container with id bb8a90c737f96aa458830e50b75b01a738cdadc216c69a937f94ba8519b46e8f Sep 16 01:29:16.814430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.814392 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:29:16.814507 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.814465 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:29:16.814507 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:16.814498 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:29:17.059919 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:17.059813 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:17.387683 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:17.387642 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" event={"ID":"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b","Type":"ContainerStarted","Data":"7159626a6b55491dc0ae0d4f669b90f723f491358faec067d571de4473c964b4"} Sep 16 01:29:17.387870 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:17.387698 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" event={"ID":"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b","Type":"ContainerStarted","Data":"bb8a90c737f96aa458830e50b75b01a738cdadc216c69a937f94ba8519b46e8f"} Sep 16 01:29:17.406847 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:17.406799 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" podStartSLOduration=1.406783585 podStartE2EDuration="1.406783585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:29:17.404009353 +0000 UTC m=+787.656431943" watchObservedRunningTime="2026-09-16 01:29:17.406783585 +0000 UTC m=+787.659206173" Sep 16 01:29:17.526312 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:17.526277 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:17.531621 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:17.531599 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:18.393495 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:18.393462 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:18.394453 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:18.394435 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:19.363192 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:19.363115 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:19.461538 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:19.461504 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:19.755126 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:19.755095 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:19.854478 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:19.854448 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:19.978262 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:19.978226 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:20.778371 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:20.778334 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:20.879581 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:20.879519 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:20.980051 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:20.979970 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:21.502865 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:21.502837 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:21.606083 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:21.606050 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:22.090518 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.090485 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:22.145596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.145558 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:22.249547 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.249480 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:22.626709 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.626677 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:22.690816 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.690783 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:22.729093 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.729063 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv"] Sep 16 01:29:22.732769 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.732743 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" Sep 16 01:29:22.735370 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.735353 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-d0c9f2-dockercfg-mgzhg\"" Sep 16 01:29:22.735445 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.735417 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-d0c9f2\"" Sep 16 01:29:22.741600 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.741150 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv"] Sep 16 01:29:22.763865 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.763836 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqvvh\" (UniqueName: \"kubernetes.io/projected/aad009cd-4750-49fb-87db-8e266b539afb-kube-api-access-xqvvh\") pod \"payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv\" (UID: \"aad009cd-4750-49fb-87db-8e266b539afb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" Sep 16 01:29:22.764048 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.764027 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad009cd-4750-49fb-87db-8e266b539afb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv\" (UID: \"aad009cd-4750-49fb-87db-8e266b539afb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" Sep 16 01:29:22.864900 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.864863 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad009cd-4750-49fb-87db-8e266b539afb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv\" (UID: \"aad009cd-4750-49fb-87db-8e266b539afb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" Sep 16 01:29:22.865062 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.864953 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqvvh\" (UniqueName: \"kubernetes.io/projected/aad009cd-4750-49fb-87db-8e266b539afb-kube-api-access-xqvvh\") pod \"payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv\" (UID: \"aad009cd-4750-49fb-87db-8e266b539afb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" Sep 16 01:29:22.865672 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.865651 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad009cd-4750-49fb-87db-8e266b539afb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv\" (UID: \"aad009cd-4750-49fb-87db-8e266b539afb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" Sep 16 01:29:22.872819 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:22.872790 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqvvh\" (UniqueName: \"kubernetes.io/projected/aad009cd-4750-49fb-87db-8e266b539afb-kube-api-access-xqvvh\") pod \"payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv\" (UID: \"aad009cd-4750-49fb-87db-8e266b539afb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" Sep 16 01:29:23.002394 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.002310 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:23.044998 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.044974 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" Sep 16 01:29:23.056341 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.056314 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:23.322460 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.322433 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv"] Sep 16 01:29:23.324178 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:29:23.324137 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad009cd_4750_49fb_87db_8e266b539afb.slice/crio-224d5232c5df561c6e14110b10cd9741aff3a4802360340f7172fe229edab0ab WatchSource:0}: Error finding container 224d5232c5df561c6e14110b10cd9741aff3a4802360340f7172fe229edab0ab: Status 404 returned error can't find the container with id 224d5232c5df561c6e14110b10cd9741aff3a4802360340f7172fe229edab0ab Sep 16 01:29:23.427462 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.427420 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" event={"ID":"aad009cd-4750-49fb-87db-8e266b539afb","Type":"ContainerStarted","Data":"224d5232c5df561c6e14110b10cd9741aff3a4802360340f7172fe229edab0ab"} Sep 16 01:29:23.446294 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.446258 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:23.547170 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.547124 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:23.547616 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.547595 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p"] Sep 16 01:29:23.552314 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.552293 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" Sep 16 01:29:23.555033 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.554909 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-033feb-dockercfg-hgrl9\"" Sep 16 01:29:23.555033 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.555025 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-033feb\"" Sep 16 01:29:23.558421 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.558399 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p"] Sep 16 01:29:23.572354 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.572297 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59c4f2e-ce71-42f2-b374-5d9e5fa910d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p\" (UID: \"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" Sep 16 01:29:23.572354 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.572323 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2ddq\" (UniqueName: \"kubernetes.io/projected/f59c4f2e-ce71-42f2-b374-5d9e5fa910d6-kube-api-access-n2ddq\") pod \"payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p\" (UID: \"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" Sep 16 01:29:23.673876 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.673824 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59c4f2e-ce71-42f2-b374-5d9e5fa910d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p\" (UID: \"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" Sep 16 01:29:23.673876 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.673872 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2ddq\" (UniqueName: \"kubernetes.io/projected/f59c4f2e-ce71-42f2-b374-5d9e5fa910d6-kube-api-access-n2ddq\") pod \"payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p\" (UID: \"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" Sep 16 01:29:23.675135 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.675104 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59c4f2e-ce71-42f2-b374-5d9e5fa910d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p\" (UID: \"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" Sep 16 01:29:23.684517 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.684489 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2ddq\" (UniqueName: \"kubernetes.io/projected/f59c4f2e-ce71-42f2-b374-5d9e5fa910d6-kube-api-access-n2ddq\") pod \"payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p\" (UID: \"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" Sep 16 01:29:23.866235 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.866211 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" Sep 16 01:29:23.871740 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:23.871709 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:24.139278 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:24.139248 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p"] Sep 16 01:29:24.140809 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:29:24.140756 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59c4f2e_ce71_42f2_b374_5d9e5fa910d6.slice/crio-b56aa3dd0d5fd95f46ffbe380438f450c53a2379d66afb1c6ad1da06ebe71056 WatchSource:0}: Error finding container b56aa3dd0d5fd95f46ffbe380438f450c53a2379d66afb1c6ad1da06ebe71056: Status 404 returned error can't find the container with id b56aa3dd0d5fd95f46ffbe380438f450c53a2379d66afb1c6ad1da06ebe71056 Sep 16 01:29:24.435827 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:24.435726 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" event={"ID":"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6","Type":"ContainerStarted","Data":"b56aa3dd0d5fd95f46ffbe380438f450c53a2379d66afb1c6ad1da06ebe71056"} Sep 16 01:29:24.556252 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:24.556211 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:25.207791 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:25.207755 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:25.443387 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:25.443349 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" event={"ID":"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6","Type":"ContainerStarted","Data":"c6b286d2464771b69ad003340e888813bc3054e89557de647fe02a1917ac7294"} Sep 16 01:29:25.443847 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:25.443543 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" Sep 16 01:29:25.445017 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:25.444992 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" event={"ID":"aad009cd-4750-49fb-87db-8e266b539afb","Type":"ContainerStarted","Data":"1f72b5889b8ae795807ca8949e3cf26b2df57e06236a341153e0840af7c011cf"} Sep 16 01:29:25.445141 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:25.445068 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" Sep 16 01:29:25.461217 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:25.461084 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" podStartSLOduration=1.840359082 podStartE2EDuration="2.461066536s" totalImagesPullingTime="620.707454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:29:25.457509156 +0000 UTC m=+795.709931758" watchObservedRunningTime="2026-09-16 01:29:25.461066536 +0000 UTC m=+795.713489125" Sep 16 01:29:25.473031 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:25.472988 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" podStartSLOduration=2.035707493 podStartE2EDuration="3.472974919s" totalImagesPullingTime="1.437267426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:29:25.471343065 +0000 UTC m=+795.723765652" watchObservedRunningTime="2026-09-16 01:29:25.472974919 +0000 UTC m=+795.725397508" Sep 16 01:29:25.720109 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:25.720018 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:29:33.557692 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:33.557654 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:33.658107 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:33.658054 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:33.801069 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:33.801000 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:34.435331 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:34.435284 2558 patch_prober.go:28] interesting pod/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 16 01:29:34.435509 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:34.435348 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 16 01:29:36.452870 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:36.452838 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" Sep 16 01:29:36.453287 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:36.452908 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" Sep 16 01:29:36.530667 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:36.530628 2558 generic.go:356] "Generic (PLEG): container finished" podID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerID="7159626a6b55491dc0ae0d4f669b90f723f491358faec067d571de4473c964b4" exitCode=0 Sep 16 01:29:36.530815 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:36.530694 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" event={"ID":"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b","Type":"ContainerDied","Data":"7159626a6b55491dc0ae0d4f669b90f723f491358faec067d571de4473c964b4"} Sep 16 01:29:36.531178 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:36.531137 2558 scope.go:117] "RemoveContainer" containerID="7159626a6b55491dc0ae0d4f669b90f723f491358faec067d571de4473c964b4" Sep 16 01:29:36.532366 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:36.532330 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:29:36.532453 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:36.532424 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:29:36.532514 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:36.532497 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:29:37.538350 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:37.538320 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" event={"ID":"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b","Type":"ContainerStarted","Data":"5572e10bc359aca4f5e46c2f9083b7b301cbaf174855830986d4ba99286d297a"} Sep 16 01:29:38.526507 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:38.526472 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:38.526871 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:38.526842 2558 patch_prober.go:28] interesting pod/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 16 01:29:38.526949 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:38.526906 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 16 01:29:39.463317 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:39.463283 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:39.526115 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:39.526086 2558 patch_prober.go:28] interesting pod/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 16 01:29:39.526305 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:39.526139 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 16 01:29:39.569278 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:39.569220 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:39.664253 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:39.664213 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:39.818381 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:39.818296 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:40.526510 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:40.526473 2558 patch_prober.go:28] interesting pod/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 16 01:29:40.526923 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:40.526538 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 16 01:29:41.526365 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:41.526329 2558 patch_prober.go:28] interesting pod/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 16 01:29:41.526553 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:41.526391 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 16 01:29:41.957342 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:41.957285 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:42.526650 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:42.526616 2558 patch_prober.go:28] interesting pod/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 16 01:29:42.527062 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:42.526670 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 16 01:29:43.229695 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:43.229657 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:43.530805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:43.530725 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:43.531181 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:43.531131 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:43.531926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:43.531906 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:29:44.049426 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:44.049391 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:44.343201 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:44.343111 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:44.590247 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:44.590216 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:44.811687 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:44.811636 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:44.907891 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:44.907864 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:45.011511 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:45.011483 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:45.108454 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:45.108372 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:45.209115 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:45.209084 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:45.308530 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:45.308501 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:45.891010 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:45.890976 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:46.206508 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:46.206433 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:46.306761 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:46.306733 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:46.766994 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:46.766957 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:47.477560 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:47.477531 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:48.622177 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.622138 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8"] Sep 16 01:29:48.625653 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.625631 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.628574 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.628545 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-482177\"/\"kube-root-ca.crt\"" Sep 16 01:29:48.628939 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.628775 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-482177\"/\"default-dockercfg-ttnmg\"" Sep 16 01:29:48.628939 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.628652 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-482177\"/\"simulator-w0-482177-kserve-self-signed-certs\"" Sep 16 01:29:48.628939 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.628721 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-482177\"/\"openshift-service-ca.crt\"" Sep 16 01:29:48.636675 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.636649 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8"] Sep 16 01:29:48.648511 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.648487 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:48.705951 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.705923 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-model-cache\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.706095 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.705959 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-tmp-dir\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.706095 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.705980 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-dshm\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.706182 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.706114 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-home\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.706222 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.706208 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8390c3ea-b357-4597-b2a2-8aea50a0b198-tls-certs\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.706257 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.706233 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br4bh\" (UniqueName: \"kubernetes.io/projected/8390c3ea-b357-4597-b2a2-8aea50a0b198-kube-api-access-br4bh\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.806832 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.806801 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-home\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.806969 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.806855 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8390c3ea-b357-4597-b2a2-8aea50a0b198-tls-certs\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.806969 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.806876 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br4bh\" (UniqueName: \"kubernetes.io/projected/8390c3ea-b357-4597-b2a2-8aea50a0b198-kube-api-access-br4bh\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.806969 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.806908 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-model-cache\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.806969 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.806924 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-tmp-dir\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.806969 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.806942 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-dshm\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.807308 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.807282 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-home\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.807308 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.807298 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-model-cache\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.807432 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.807373 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-tmp-dir\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.809491 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.809473 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-dshm\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.809616 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.809594 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8390c3ea-b357-4597-b2a2-8aea50a0b198-tls-certs\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.814457 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.814433 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br4bh\" (UniqueName: \"kubernetes.io/projected/8390c3ea-b357-4597-b2a2-8aea50a0b198-kube-api-access-br4bh\") pod \"simulator-w0-482177-kserve-6d769b855b-sf7h8\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:48.949062 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:48.948978 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:49.220897 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:49.220870 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8"] Sep 16 01:29:49.223433 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:29:49.223392 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8390c3ea_b357_4597_b2a2_8aea50a0b198.slice/crio-e1e698f2b804f2d2390445a256c63cd2aa1319b686a103e6ac10aecbb6c0b8ca WatchSource:0}: Error finding container e1e698f2b804f2d2390445a256c63cd2aa1319b686a103e6ac10aecbb6c0b8ca: Status 404 returned error can't find the container with id e1e698f2b804f2d2390445a256c63cd2aa1319b686a103e6ac10aecbb6c0b8ca Sep 16 01:29:49.616344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:49.616300 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" event={"ID":"8390c3ea-b357-4597-b2a2-8aea50a0b198","Type":"ContainerStarted","Data":"6c76960189e69fc1546fec54b76350de5a53b9059984400c5fadae917d10056a"} Sep 16 01:29:49.616501 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:49.616357 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" event={"ID":"8390c3ea-b357-4597-b2a2-8aea50a0b198","Type":"ContainerStarted","Data":"e1e698f2b804f2d2390445a256c63cd2aa1319b686a103e6ac10aecbb6c0b8ca"} Sep 16 01:29:49.633217 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:49.633175 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" podStartSLOduration=1.633139981 podStartE2EDuration="1.633139981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:29:49.632427517 +0000 UTC m=+819.884850110" watchObservedRunningTime="2026-09-16 01:29:49.633139981 +0000 UTC m=+819.885562933" Sep 16 01:29:49.782276 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:49.782239 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:50.117040 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.117010 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:50.405121 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.405035 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q"] Sep 16 01:29:50.408829 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.408805 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.410822 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.410801 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a58460-openshift-default-dockercfg-j7m87\"" Sep 16 01:29:50.426132 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.426079 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q"] Sep 16 01:29:50.435083 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.435058 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:50.522541 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.522508 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istiod-ca-cert\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.522697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.522564 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-token\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.522697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.522603 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-workload-socket\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.522697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.522620 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-envoy\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.522697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.522654 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-workload-certs\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.522697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.522679 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-podinfo\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.522919 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.522704 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-data\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.522919 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.522719 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxk2n\" (UniqueName: \"kubernetes.io/projected/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-kube-api-access-bxk2n\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.522919 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.522749 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-credential-socket\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.539770 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.539742 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:50.623861 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.623834 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-credential-socket\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.624028 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.623883 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istiod-ca-cert\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.624028 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.623919 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-token\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.624028 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.623949 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-workload-socket\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.624028 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.623966 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-envoy\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.624028 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.624013 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-workload-certs\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.624363 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.624040 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-podinfo\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.624363 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.624067 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-data\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.624363 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.624095 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxk2n\" (UniqueName: \"kubernetes.io/projected/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-kube-api-access-bxk2n\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.624363 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.624333 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-credential-socket\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.624581 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.624380 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-workload-certs\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.624581 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.624532 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-data\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.624661 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.624612 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-workload-socket\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.624755 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.624737 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istiod-ca-cert\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.626528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.626504 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-envoy\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.626714 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.626693 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-podinfo\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.630826 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.630804 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-token\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.630909 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.630811 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxk2n\" (UniqueName: \"kubernetes.io/projected/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-kube-api-access-bxk2n\") pod \"e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:50.719639 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:50.719574 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:51.001048 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:51.001019 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q"] Sep 16 01:29:51.002236 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:29:51.002211 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9db5a2_7b97_43b2_8fa1_116d8cf2abea.slice/crio-f8e059b5e214fbade6d6fd9462c16b592b10ce49284db42c86796f0ee4109412 WatchSource:0}: Error finding container f8e059b5e214fbade6d6fd9462c16b592b10ce49284db42c86796f0ee4109412: Status 404 returned error can't find the container with id f8e059b5e214fbade6d6fd9462c16b592b10ce49284db42c86796f0ee4109412 Sep 16 01:29:51.005151 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:51.005113 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:29:51.005231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:51.005214 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:29:51.005277 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:51.005261 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:29:51.629983 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:51.629951 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" event={"ID":"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea","Type":"ContainerStarted","Data":"9652ad4f0ed457f913ce019d0041833b3f6776760f4a1c18da683ae4210cd937"} Sep 16 01:29:51.630186 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:51.629998 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" event={"ID":"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea","Type":"ContainerStarted","Data":"f8e059b5e214fbade6d6fd9462c16b592b10ce49284db42c86796f0ee4109412"} Sep 16 01:29:51.650097 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:51.650051 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" podStartSLOduration=1.6500350830000001 podStartE2EDuration="1.650035083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:29:51.64701514 +0000 UTC m=+821.899437729" watchObservedRunningTime="2026-09-16 01:29:51.650035083 +0000 UTC m=+821.902457671" Sep 16 01:29:51.720512 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:51.720485 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:52.725663 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:52.725633 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:52.954868 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:52.954828 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:53.120113 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:53.120075 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:29:53.582646 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:53.582567 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:53.641337 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:53.641308 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:53.642261 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:53.642245 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:29:53.669415 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:53.669383 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:54.753979 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:54.753946 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:55.381366 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:55.381331 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:56.166471 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.166438 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:56.267004 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.266973 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:56.599897 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.599827 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:56.699365 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.699333 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:56.706020 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.705996 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr"] Sep 16 01:29:56.709841 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.709823 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" Sep 16 01:29:56.711809 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.711788 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a58460\"" Sep 16 01:29:56.711930 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.711870 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a58460-dockercfg-s77v9\"" Sep 16 01:29:56.720422 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.720400 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr"] Sep 16 01:29:56.745543 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.742444 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t"] Sep 16 01:29:56.748879 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.747752 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" Sep 16 01:29:56.754092 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.754066 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t"] Sep 16 01:29:56.784943 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.784914 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z862p\" (UniqueName: \"kubernetes.io/projected/71958bb4-b1b5-402b-aa36-7beec61f5829-kube-api-access-z862p\") pod \"payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr\" (UID: \"71958bb4-b1b5-402b-aa36-7beec61f5829\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" Sep 16 01:29:56.785069 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.785022 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71958bb4-b1b5-402b-aa36-7beec61f5829-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr\" (UID: \"71958bb4-b1b5-402b-aa36-7beec61f5829\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" Sep 16 01:29:56.886574 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.886540 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71958bb4-b1b5-402b-aa36-7beec61f5829-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr\" (UID: \"71958bb4-b1b5-402b-aa36-7beec61f5829\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" Sep 16 01:29:56.886739 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.886594 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d5ee9f-125b-4074-8ec2-ad0b92df1009-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t\" (UID: \"47d5ee9f-125b-4074-8ec2-ad0b92df1009\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" Sep 16 01:29:56.886739 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.886638 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z862p\" (UniqueName: \"kubernetes.io/projected/71958bb4-b1b5-402b-aa36-7beec61f5829-kube-api-access-z862p\") pod \"payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr\" (UID: \"71958bb4-b1b5-402b-aa36-7beec61f5829\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" Sep 16 01:29:56.886739 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.886660 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxd7w\" (UniqueName: \"kubernetes.io/projected/47d5ee9f-125b-4074-8ec2-ad0b92df1009-kube-api-access-bxd7w\") pod \"payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t\" (UID: \"47d5ee9f-125b-4074-8ec2-ad0b92df1009\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" Sep 16 01:29:56.887256 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.887236 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71958bb4-b1b5-402b-aa36-7beec61f5829-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr\" (UID: \"71958bb4-b1b5-402b-aa36-7beec61f5829\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" Sep 16 01:29:56.894032 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.894006 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z862p\" (UniqueName: \"kubernetes.io/projected/71958bb4-b1b5-402b-aa36-7beec61f5829-kube-api-access-z862p\") pod \"payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr\" (UID: \"71958bb4-b1b5-402b-aa36-7beec61f5829\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" Sep 16 01:29:56.962809 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.962775 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:56.987961 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.987932 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxd7w\" (UniqueName: \"kubernetes.io/projected/47d5ee9f-125b-4074-8ec2-ad0b92df1009-kube-api-access-bxd7w\") pod \"payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t\" (UID: \"47d5ee9f-125b-4074-8ec2-ad0b92df1009\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" Sep 16 01:29:56.988073 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.988021 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d5ee9f-125b-4074-8ec2-ad0b92df1009-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t\" (UID: \"47d5ee9f-125b-4074-8ec2-ad0b92df1009\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" Sep 16 01:29:56.988677 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.988652 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d5ee9f-125b-4074-8ec2-ad0b92df1009-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t\" (UID: \"47d5ee9f-125b-4074-8ec2-ad0b92df1009\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" Sep 16 01:29:56.995271 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:56.995249 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxd7w\" (UniqueName: \"kubernetes.io/projected/47d5ee9f-125b-4074-8ec2-ad0b92df1009-kube-api-access-bxd7w\") pod \"payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t\" (UID: \"47d5ee9f-125b-4074-8ec2-ad0b92df1009\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" Sep 16 01:29:57.022027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:57.022001 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" Sep 16 01:29:57.059770 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:57.059730 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" Sep 16 01:29:57.300539 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:57.300467 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr"] Sep 16 01:29:57.303735 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:29:57.303710 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71958bb4_b1b5_402b_aa36_7beec61f5829.slice/crio-a75eff5bd3a9a576ae6950ba6e2599edd52f7b7c9aca5bd80358d18cc8cd39bb WatchSource:0}: Error finding container a75eff5bd3a9a576ae6950ba6e2599edd52f7b7c9aca5bd80358d18cc8cd39bb: Status 404 returned error can't find the container with id a75eff5bd3a9a576ae6950ba6e2599edd52f7b7c9aca5bd80358d18cc8cd39bb Sep 16 01:29:57.344668 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:57.344648 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t"] Sep 16 01:29:57.345723 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:29:57.345696 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d5ee9f_125b_4074_8ec2_ad0b92df1009.slice/crio-f71b6983524c376f849336f8997f6aa0857359487473a8f313d8687dbd34d56d WatchSource:0}: Error finding container f71b6983524c376f849336f8997f6aa0857359487473a8f313d8687dbd34d56d: Status 404 returned error can't find the container with id f71b6983524c376f849336f8997f6aa0857359487473a8f313d8687dbd34d56d Sep 16 01:29:57.666608 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:57.666570 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" event={"ID":"71958bb4-b1b5-402b-aa36-7beec61f5829","Type":"ContainerStarted","Data":"21d34ec313e2d6bb3eea6b42a9df3bee6f7b20ee584bfd1c138477e4e4955538"} Sep 16 01:29:57.666799 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:57.666643 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" Sep 16 01:29:57.666799 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:57.666671 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" event={"ID":"71958bb4-b1b5-402b-aa36-7beec61f5829","Type":"ContainerStarted","Data":"a75eff5bd3a9a576ae6950ba6e2599edd52f7b7c9aca5bd80358d18cc8cd39bb"} Sep 16 01:29:57.668559 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:57.668532 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" event={"ID":"47d5ee9f-125b-4074-8ec2-ad0b92df1009","Type":"ContainerStarted","Data":"6d627e850e90daeee28f1ddd21831ec6142a37f2f79e2dd98efd267f7588e4cc"} Sep 16 01:29:57.668698 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:57.668589 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" Sep 16 01:29:57.668698 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:57.668612 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" event={"ID":"47d5ee9f-125b-4074-8ec2-ad0b92df1009","Type":"ContainerStarted","Data":"f71b6983524c376f849336f8997f6aa0857359487473a8f313d8687dbd34d56d"} Sep 16 01:29:57.684534 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:57.684482 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" podStartSLOduration=1.6844668189999998 podStartE2EDuration="1.684466819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:29:57.681281433 +0000 UTC m=+827.933704023" watchObservedRunningTime="2026-09-16 01:29:57.684466819 +0000 UTC m=+827.936889457" Sep 16 01:29:57.699269 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:57.699230 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" podStartSLOduration=1.699217082 podStartE2EDuration="1.699217082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:29:57.696054633 +0000 UTC m=+827.948477224" watchObservedRunningTime="2026-09-16 01:29:57.699217082 +0000 UTC m=+827.951639671" Sep 16 01:29:57.846954 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:57.846917 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:29:58.949968 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:58.949938 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:58.950455 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:58.949990 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:58.964361 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:58.964336 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:29:59.692454 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:29:59.692424 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:30:00.138383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:00.138350 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k"] Sep 16 01:30:00.142444 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:00.142421 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn"] Sep 16 01:30:00.142563 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:00.142525 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k" Sep 16 01:30:00.144497 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:00.144478 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4ljjd\"" Sep 16 01:30:00.145831 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:00.145806 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn" Sep 16 01:30:00.149407 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:00.149385 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k"] Sep 16 01:30:00.152668 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:00.152493 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn"] Sep 16 01:30:00.158321 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:00.158302 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k" Sep 16 01:30:00.167723 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:00.167702 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn" Sep 16 01:30:00.451296 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:00.451267 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k"] Sep 16 01:30:00.452664 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:30:00.452634 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52963ff_75a9_456c_b9e6_835f7310496a.slice/crio-0d2c26fb60c35f57c80fc8fd5fa39bfab8f2c8e968ad4f0d10b3651a25812862 WatchSource:0}: Error finding container 0d2c26fb60c35f57c80fc8fd5fa39bfab8f2c8e968ad4f0d10b3651a25812862: Status 404 returned error can't find the container with id 0d2c26fb60c35f57c80fc8fd5fa39bfab8f2c8e968ad4f0d10b3651a25812862 Sep 16 01:30:00.457210 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:00.457187 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn"] Sep 16 01:30:00.458535 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:30:00.458514 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5af05c9_f1cb_455a_b915_5ab99dca0f52.slice/crio-c5dee32dec4d936d3a6725f3b43b990d433b47ac277bf51dfb0c960c46974f67 WatchSource:0}: Error finding container c5dee32dec4d936d3a6725f3b43b990d433b47ac277bf51dfb0c960c46974f67: Status 404 returned error can't find the container with id c5dee32dec4d936d3a6725f3b43b990d433b47ac277bf51dfb0c960c46974f67 Sep 16 01:30:00.687647 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:00.687613 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn" event={"ID":"c5af05c9-f1cb-455a-b915-5ab99dca0f52","Type":"ContainerStarted","Data":"c5dee32dec4d936d3a6725f3b43b990d433b47ac277bf51dfb0c960c46974f67"} Sep 16 01:30:00.688893 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:00.688868 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k" event={"ID":"d52963ff-75a9-456c-b9e6-835f7310496a","Type":"ContainerStarted","Data":"0d2c26fb60c35f57c80fc8fd5fa39bfab8f2c8e968ad4f0d10b3651a25812862"} Sep 16 01:30:03.711807 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:03.711701 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn" event={"ID":"c5af05c9-f1cb-455a-b915-5ab99dca0f52","Type":"ContainerStarted","Data":"e4ba8113cedb93864c4baafdee4ae643f2a53236eb3829abecf4c067b0bed6ad"} Sep 16 01:30:03.713238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:03.713189 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k" event={"ID":"d52963ff-75a9-456c-b9e6-835f7310496a","Type":"ContainerStarted","Data":"8d9ec735819be6c52cf9bda6912dacc55ae6726829c8052fc2bff6d5181c9ff3"} Sep 16 01:30:03.726138 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:03.726092 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn" podStartSLOduration=1.122694249 podStartE2EDuration="3.726077137s" totalImagesPullingTime="2.603382888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:30:03.723840523 +0000 UTC m=+833.976263113" watchObservedRunningTime="2026-09-16 01:30:03.726077137 +0000 UTC m=+833.978499726" Sep 16 01:30:03.737184 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:03.737128 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k" podStartSLOduration=1.118281568 podStartE2EDuration="3.737118473s" totalImagesPullingTime="2.618836905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:30:03.736216582 +0000 UTC m=+833.988639172" watchObservedRunningTime="2026-09-16 01:30:03.737118473 +0000 UTC m=+833.989541062" Sep 16 01:30:04.721475 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:04.721440 2558 generic.go:356] "Generic (PLEG): container finished" podID="c5af05c9-f1cb-455a-b915-5ab99dca0f52" containerID="e4ba8113cedb93864c4baafdee4ae643f2a53236eb3829abecf4c067b0bed6ad" exitCode=7 Sep 16 01:30:04.721918 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:04.721516 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn" event={"ID":"c5af05c9-f1cb-455a-b915-5ab99dca0f52","Type":"ContainerDied","Data":"e4ba8113cedb93864c4baafdee4ae643f2a53236eb3829abecf4c067b0bed6ad"} Sep 16 01:30:04.721918 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:04.721679 2558 scope.go:117] "RemoveContainer" containerID="e4ba8113cedb93864c4baafdee4ae643f2a53236eb3829abecf4c067b0bed6ad" Sep 16 01:30:04.723202 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:04.723176 2558 generic.go:356] "Generic (PLEG): container finished" podID="d52963ff-75a9-456c-b9e6-835f7310496a" containerID="8d9ec735819be6c52cf9bda6912dacc55ae6726829c8052fc2bff6d5181c9ff3" exitCode=0 Sep 16 01:30:04.723309 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:04.723256 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k" event={"ID":"d52963ff-75a9-456c-b9e6-835f7310496a","Type":"ContainerDied","Data":"8d9ec735819be6c52cf9bda6912dacc55ae6726829c8052fc2bff6d5181c9ff3"} Sep 16 01:30:05.731842 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:05.731805 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn" event={"ID":"c5af05c9-f1cb-455a-b915-5ab99dca0f52","Type":"ContainerStarted","Data":"b2ee7065cfa95f29b37bcc42dc2bf7dddd6213d0469c8ac5a1d21e2026bfe68c"} Sep 16 01:30:05.919691 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:05.919666 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k" Sep 16 01:30:06.739693 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:06.739659 2558 generic.go:356] "Generic (PLEG): container finished" podID="c5af05c9-f1cb-455a-b915-5ab99dca0f52" containerID="b2ee7065cfa95f29b37bcc42dc2bf7dddd6213d0469c8ac5a1d21e2026bfe68c" exitCode=0 Sep 16 01:30:06.740141 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:06.739692 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn" event={"ID":"c5af05c9-f1cb-455a-b915-5ab99dca0f52","Type":"ContainerDied","Data":"b2ee7065cfa95f29b37bcc42dc2bf7dddd6213d0469c8ac5a1d21e2026bfe68c"} Sep 16 01:30:06.740141 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:06.739749 2558 scope.go:117] "RemoveContainer" containerID="e4ba8113cedb93864c4baafdee4ae643f2a53236eb3829abecf4c067b0bed6ad" Sep 16 01:30:06.741222 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:06.741207 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k" Sep 16 01:30:06.741323 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:06.741274 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k" event={"ID":"d52963ff-75a9-456c-b9e6-835f7310496a","Type":"ContainerDied","Data":"0d2c26fb60c35f57c80fc8fd5fa39bfab8f2c8e968ad4f0d10b3651a25812862"} Sep 16 01:30:06.741364 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:06.741327 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d2c26fb60c35f57c80fc8fd5fa39bfab8f2c8e968ad4f0d10b3651a25812862" Sep 16 01:30:07.934528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:07.934506 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn" Sep 16 01:30:08.679318 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:08.679277 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" Sep 16 01:30:08.679526 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:08.679399 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" Sep 16 01:30:08.757196 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:08.757143 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn" Sep 16 01:30:08.757196 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:08.757181 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn" event={"ID":"c5af05c9-f1cb-455a-b915-5ab99dca0f52","Type":"ContainerDied","Data":"c5dee32dec4d936d3a6725f3b43b990d433b47ac277bf51dfb0c960c46974f67"} Sep 16 01:30:08.757396 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:08.757229 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5dee32dec4d936d3a6725f3b43b990d433b47ac277bf51dfb0c960c46974f67" Sep 16 01:30:09.968484 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:09.968026 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:30:11.264704 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:11.264656 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:30:11.809678 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:11.809643 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:30:12.469019 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:12.468986 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:30:12.785842 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:12.785761 2558 generic.go:356] "Generic (PLEG): container finished" podID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerID="9652ad4f0ed457f913ce019d0041833b3f6776760f4a1c18da683ae4210cd937" exitCode=0 Sep 16 01:30:12.785986 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:12.785831 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" event={"ID":"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea","Type":"ContainerDied","Data":"9652ad4f0ed457f913ce019d0041833b3f6776760f4a1c18da683ae4210cd937"} Sep 16 01:30:12.786261 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:12.786245 2558 scope.go:117] "RemoveContainer" containerID="9652ad4f0ed457f913ce019d0041833b3f6776760f4a1c18da683ae4210cd937" Sep 16 01:30:12.787485 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:12.787446 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:30:12.787616 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:12.787540 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:30:12.787616 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:12.787587 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:30:13.799654 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:13.799614 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" event={"ID":"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea","Type":"ContainerStarted","Data":"324927dfa6df7beb692965e14fa025ef27a2e11a6c0d3980c6180193fe32c71b"} Sep 16 01:30:14.720304 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:14.720271 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:30:14.720493 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:14.720429 2558 patch_prober.go:28] interesting pod/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 16 01:30:14.720565 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:14.720484 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" podUID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 16 01:30:15.720278 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:15.720244 2558 patch_prober.go:28] interesting pod/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 16 01:30:15.720747 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:15.720297 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" podUID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 16 01:30:16.720129 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:16.720092 2558 patch_prober.go:28] interesting pod/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 16 01:30:16.720349 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:16.720149 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" podUID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 16 01:30:17.720537 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:17.720503 2558 patch_prober.go:28] interesting pod/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 16 01:30:17.720925 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:17.720554 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" podUID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 16 01:30:18.059213 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:18.059086 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:30:18.606823 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:18.606791 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:30:18.719917 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:18.719885 2558 patch_prober.go:28] interesting pod/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 16 01:30:18.720091 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:18.719943 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" podUID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 16 01:30:18.785882 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:18.785840 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:30:19.396712 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:19.396676 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:30:19.723844 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:19.723768 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:30:19.724151 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:19.724131 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:30:19.724751 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:19.724733 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:30:20.155740 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:20.155704 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:30:25.116726 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:25.116677 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:30:35.349337 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:35.349303 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:30:40.015460 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:40.015425 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:30:52.612804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:52.612770 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:30:53.516410 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.516380 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2"] Sep 16 01:30:53.516889 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.516871 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5af05c9-f1cb-455a-b915-5ab99dca0f52" containerName="cleanup" Sep 16 01:30:53.516991 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.516892 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5af05c9-f1cb-455a-b915-5ab99dca0f52" containerName="cleanup" Sep 16 01:30:53.516991 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.516906 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5af05c9-f1cb-455a-b915-5ab99dca0f52" containerName="cleanup" Sep 16 01:30:53.516991 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.516914 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5af05c9-f1cb-455a-b915-5ab99dca0f52" containerName="cleanup" Sep 16 01:30:53.516991 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.516944 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52963ff-75a9-456c-b9e6-835f7310496a" containerName="cleanup" Sep 16 01:30:53.516991 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.516952 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52963ff-75a9-456c-b9e6-835f7310496a" containerName="cleanup" Sep 16 01:30:53.517280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.517053 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5af05c9-f1cb-455a-b915-5ab99dca0f52" containerName="cleanup" Sep 16 01:30:53.517280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.517066 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5af05c9-f1cb-455a-b915-5ab99dca0f52" containerName="cleanup" Sep 16 01:30:53.517280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.517074 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52963ff-75a9-456c-b9e6-835f7310496a" containerName="cleanup" Sep 16 01:30:53.520402 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.520382 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.522560 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.522536 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-482177\"/\"premium-w0-482177-kserve-self-signed-certs\"" Sep 16 01:30:53.527150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.527119 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2"] Sep 16 01:30:53.538638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.538618 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:30:53.542907 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.542884 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbslp\" (UniqueName: \"kubernetes.io/projected/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-kube-api-access-tbslp\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.542973 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.542917 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-model-cache\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.542973 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.542961 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-home\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.543113 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.543087 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-tls-certs\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.543188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.543149 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-dshm\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.543227 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.543207 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-tmp-dir\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.643695 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.643655 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-dshm\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.643695 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.643699 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-tmp-dir\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.644513 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.643784 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbslp\" (UniqueName: \"kubernetes.io/projected/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-kube-api-access-tbslp\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.644513 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.643805 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-model-cache\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.644513 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.643850 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-home\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.644513 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.643918 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-tls-certs\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.644513 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.644281 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-model-cache\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.644513 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.644324 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-home\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.644745 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.644501 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-tmp-dir\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.646186 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.646138 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-dshm\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.646814 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.646797 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-tls-certs\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.651371 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.651349 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbslp\" (UniqueName: \"kubernetes.io/projected/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-kube-api-access-tbslp\") pod \"premium-w0-482177-kserve-f77675fd9-dcsl2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:53.832315 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:53.832237 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:30:54.105694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:54.105663 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2"] Sep 16 01:30:54.107267 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:30:54.107241 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ba6359e_f8bc_44f4_88c0_19e22ed5d7e2.slice/crio-a0da232d53f7fec872d4330b77f4c2bfdfa4511602e164b323a7c8ac86ea8853 WatchSource:0}: Error finding container a0da232d53f7fec872d4330b77f4c2bfdfa4511602e164b323a7c8ac86ea8853: Status 404 returned error can't find the container with id a0da232d53f7fec872d4330b77f4c2bfdfa4511602e164b323a7c8ac86ea8853 Sep 16 01:30:55.046220 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:55.046181 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" event={"ID":"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2","Type":"ContainerStarted","Data":"f2e1f7f6ab52430ba8b34378c642eb89a2aea988496bfb90f15b5a17aafb6a99"} Sep 16 01:30:55.046591 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:55.046234 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" event={"ID":"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2","Type":"ContainerStarted","Data":"a0da232d53f7fec872d4330b77f4c2bfdfa4511602e164b323a7c8ac86ea8853"} Sep 16 01:30:55.062794 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:30:55.062739 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" podStartSLOduration=2.062721426 podStartE2EDuration="2.062721426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:30:55.060150243 +0000 UTC m=+885.312572826" watchObservedRunningTime="2026-09-16 01:30:55.062721426 +0000 UTC m=+885.315144015" Sep 16 01:31:03.833046 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:03.832959 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:31:03.833604 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:03.833041 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:31:03.845886 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:03.845862 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:31:04.112674 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:04.112597 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:31:06.212891 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:06.212854 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:31:10.249096 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:10.249061 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:31:10.250258 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:10.250237 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:31:33.322172 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.322123 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27"] Sep 16 01:31:33.326224 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.326201 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.328441 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.328413 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d0c9f2\"/\"openshift-service-ca.crt\"" Sep 16 01:31:33.328557 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.328419 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d0c9f2\"/\"premium-w3-d0c9f2-kserve-self-signed-certs\"" Sep 16 01:31:33.329775 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.329755 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d0c9f2\"/\"default-dockercfg-8n69k\"" Sep 16 01:31:33.330048 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.329785 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d0c9f2\"/\"kube-root-ca.crt\"" Sep 16 01:31:33.334649 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.334627 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27"] Sep 16 01:31:33.349092 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.349064 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:31:33.509120 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.509085 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-dshm\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.509120 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.509127 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-home\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.509384 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.509249 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-tmp-dir\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.509384 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.509287 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zvtr\" (UniqueName: \"kubernetes.io/projected/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-kube-api-access-5zvtr\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.509384 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.509352 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-model-cache\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.509506 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.509410 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-tls-certs\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.610750 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.610650 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-tls-certs\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.610750 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.610714 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-dshm\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.610750 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.610741 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-home\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.611102 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.610784 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-tmp-dir\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.611102 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.610799 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvtr\" (UniqueName: \"kubernetes.io/projected/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-kube-api-access-5zvtr\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.611102 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.610830 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-model-cache\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.611297 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.611219 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-home\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.611297 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.611279 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-tmp-dir\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.611368 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.611303 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-model-cache\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.613317 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.613290 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-dshm\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.613533 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.613514 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-tls-certs\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.618215 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.618194 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zvtr\" (UniqueName: \"kubernetes.io/projected/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-kube-api-access-5zvtr\") pod \"premium-w3-d0c9f2-kserve-6774c46b69-94w27\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.638243 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.638220 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:33.920051 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:33.920019 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27"] Sep 16 01:31:33.921530 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:31:33.921502 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5cb3eea_9abe_4b45_a8a1_3d0afca700d0.slice/crio-b00c8c3c624fac676cff9fd8781e7f64a16b825b4882b5d71eaf883d89f973e9 WatchSource:0}: Error finding container b00c8c3c624fac676cff9fd8781e7f64a16b825b4882b5d71eaf883d89f973e9: Status 404 returned error can't find the container with id b00c8c3c624fac676cff9fd8781e7f64a16b825b4882b5d71eaf883d89f973e9 Sep 16 01:31:34.295503 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:34.295426 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" event={"ID":"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0","Type":"ContainerStarted","Data":"b6f90d764a6cbab2b6beed9f998a33d37c0c29d829de72be853d024ec6830cc7"} Sep 16 01:31:34.295503 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:34.295463 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" event={"ID":"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0","Type":"ContainerStarted","Data":"b00c8c3c624fac676cff9fd8781e7f64a16b825b4882b5d71eaf883d89f973e9"} Sep 16 01:31:34.306322 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:34.306282 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" podStartSLOduration=1.306266779 podStartE2EDuration="1.306266779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:31:34.303967969 +0000 UTC m=+924.556390560" watchObservedRunningTime="2026-09-16 01:31:34.306266779 +0000 UTC m=+924.558689421" Sep 16 01:31:39.123746 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:39.123711 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:31:43.638362 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:43.638321 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:43.638812 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:43.638576 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:43.651173 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:43.651131 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:31:44.358853 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:31:44.358825 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:32:00.612350 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:32:00.612313 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:32:16.828309 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:32:16.828267 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:32:22.624402 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:32:22.624369 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:32:50.753005 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:32:50.752970 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:32:57.917379 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:32:57.917345 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:33:16.783044 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:16.783008 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:16.882702 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:16.882666 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:17.334081 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:17.334048 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:22.818274 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:22.818231 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:22.918973 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:22.918944 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:23.085534 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:23.085449 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:23.405847 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:23.405815 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:23.620922 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:23.620846 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:24.013732 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:24.013697 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:33:25.660536 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:25.660501 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:26.298181 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:26.298134 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:26.405809 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:26.405751 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:26.561037 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:26.560955 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:27.294486 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.294446 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:27.403678 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.403645 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9"] Sep 16 01:33:27.407504 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.407477 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.409556 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.409534 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-482177\"/\"distinct-w0-482177-kserve-self-signed-certs\"" Sep 16 01:33:27.416241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.416221 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9"] Sep 16 01:33:27.556092 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.556013 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm5tq\" (UniqueName: \"kubernetes.io/projected/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-kube-api-access-pm5tq\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.556092 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.556059 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-tls-certs\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.556366 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.556137 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-model-cache\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.556366 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.556199 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-home\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.556366 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.556236 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-dshm\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.556366 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.556336 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-tmp-dir\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.657364 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.657333 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-tls-certs\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.657519 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.657393 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-model-cache\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.657588 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.657510 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-home\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.657588 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.657567 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-dshm\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.657704 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.657646 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-tmp-dir\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.657760 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.657704 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm5tq\" (UniqueName: \"kubernetes.io/projected/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-kube-api-access-pm5tq\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.657760 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.657711 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-model-cache\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.657860 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.657837 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-home\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.658008 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.657984 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-tmp-dir\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.659967 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.659936 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-dshm\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.660093 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.660038 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-tls-certs\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.664942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.664918 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm5tq\" (UniqueName: \"kubernetes.io/projected/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-kube-api-access-pm5tq\") pod \"distinct-w0-482177-kserve-c9bc568d-rpxh9\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.717931 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.717904 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:27.904943 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.904908 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:27.994917 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.994885 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9"] Sep 16 01:33:27.996101 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:27.996078 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:27.996880 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:33:27.996853 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea1423a7_3e9e_4ed2_ae2d_62530a212c19.slice/crio-889d41ceca4e3dd698fdec77192154674a2c706ed2aabcaf9c435b2a9557aa01 WatchSource:0}: Error finding container 889d41ceca4e3dd698fdec77192154674a2c706ed2aabcaf9c435b2a9557aa01: Status 404 returned error can't find the container with id 889d41ceca4e3dd698fdec77192154674a2c706ed2aabcaf9c435b2a9557aa01 Sep 16 01:33:28.648569 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:28.648527 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:28.989031 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:28.988959 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" event={"ID":"ea1423a7-3e9e-4ed2-ae2d-62530a212c19","Type":"ContainerStarted","Data":"120e17f9fd5d9aea87c607f0998d62d60d6995999db88d252234f28255bc668e"} Sep 16 01:33:28.989031 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:28.988997 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" event={"ID":"ea1423a7-3e9e-4ed2-ae2d-62530a212c19","Type":"ContainerStarted","Data":"889d41ceca4e3dd698fdec77192154674a2c706ed2aabcaf9c435b2a9557aa01"} Sep 16 01:33:29.003734 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:29.003691 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" podStartSLOduration=2.003676447 podStartE2EDuration="2.003676447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:33:29.002000254 +0000 UTC m=+1039.254422854" watchObservedRunningTime="2026-09-16 01:33:29.003676447 +0000 UTC m=+1039.256099036" Sep 16 01:33:29.269792 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:29.269710 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:29.323731 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:29.323705 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:29.691385 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:29.691348 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:29.792349 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:29.792318 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:29.832906 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:29.832873 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4"] Sep 16 01:33:29.836568 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:29.836548 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" Sep 16 01:33:29.841392 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:29.841331 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ba321c07-dockercfg-z4twl\"" Sep 16 01:33:29.841898 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:29.841881 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ba321c07\"" Sep 16 01:33:29.850005 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:29.849987 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4"] Sep 16 01:33:29.982402 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:29.982321 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df5c422-e9d8-4c0b-b645-8276164d00b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-ba321c07-549495cf89-n5df4\" (UID: \"3df5c422-e9d8-4c0b-b645-8276164d00b9\") " pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" Sep 16 01:33:29.982402 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:29.982360 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-594xz\" (UniqueName: \"kubernetes.io/projected/3df5c422-e9d8-4c0b-b645-8276164d00b9-kube-api-access-594xz\") pod \"payload-processing-e2e-ait-ba321c07-549495cf89-n5df4\" (UID: \"3df5c422-e9d8-4c0b-b645-8276164d00b9\") " pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" Sep 16 01:33:30.083570 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:30.083537 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df5c422-e9d8-4c0b-b645-8276164d00b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-ba321c07-549495cf89-n5df4\" (UID: \"3df5c422-e9d8-4c0b-b645-8276164d00b9\") " pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" Sep 16 01:33:30.083570 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:30.083574 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-594xz\" (UniqueName: \"kubernetes.io/projected/3df5c422-e9d8-4c0b-b645-8276164d00b9-kube-api-access-594xz\") pod \"payload-processing-e2e-ait-ba321c07-549495cf89-n5df4\" (UID: \"3df5c422-e9d8-4c0b-b645-8276164d00b9\") " pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" Sep 16 01:33:30.084363 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:30.084344 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df5c422-e9d8-4c0b-b645-8276164d00b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-ba321c07-549495cf89-n5df4\" (UID: \"3df5c422-e9d8-4c0b-b645-8276164d00b9\") " pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" Sep 16 01:33:30.091387 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:30.091361 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-594xz\" (UniqueName: \"kubernetes.io/projected/3df5c422-e9d8-4c0b-b645-8276164d00b9-kube-api-access-594xz\") pod \"payload-processing-e2e-ait-ba321c07-549495cf89-n5df4\" (UID: \"3df5c422-e9d8-4c0b-b645-8276164d00b9\") " pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" Sep 16 01:33:30.148296 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:30.148272 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" Sep 16 01:33:30.187472 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:30.187439 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:30.426631 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:30.426607 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4"] Sep 16 01:33:30.433098 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:33:30.429561 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df5c422_e9d8_4c0b_b645_8276164d00b9.slice/crio-ff0676aa557a5c034aefbc72218938ed273b8785c460bfc6c1a7330a5d67a239 WatchSource:0}: Error finding container ff0676aa557a5c034aefbc72218938ed273b8785c460bfc6c1a7330a5d67a239: Status 404 returned error can't find the container with id ff0676aa557a5c034aefbc72218938ed273b8785c460bfc6c1a7330a5d67a239 Sep 16 01:33:31.004843 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:31.004802 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" event={"ID":"3df5c422-e9d8-4c0b-b645-8276164d00b9","Type":"ContainerStarted","Data":"12c9836f3177a2cb811afac3ae9ae63d6e68576d985e7fa6a1772432129759df"} Sep 16 01:33:31.005290 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:31.004880 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" event={"ID":"3df5c422-e9d8-4c0b-b645-8276164d00b9","Type":"ContainerStarted","Data":"ff0676aa557a5c034aefbc72218938ed273b8785c460bfc6c1a7330a5d67a239"} Sep 16 01:33:31.005290 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:31.005020 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" Sep 16 01:33:31.019212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:31.019151 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" podStartSLOduration=2.019137454 podStartE2EDuration="2.019137454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:33:31.01752404 +0000 UTC m=+1041.269946643" watchObservedRunningTime="2026-09-16 01:33:31.019137454 +0000 UTC m=+1041.271560046" Sep 16 01:33:37.718438 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:37.718399 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:37.718845 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:37.718471 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:37.731325 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:37.731301 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:38.059115 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:38.059031 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:33:42.013008 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:42.012978 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" Sep 16 01:33:43.456292 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:43.456262 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:49.868764 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:49.868722 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:51.790126 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:51.790091 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:52.570583 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:52.570550 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:53.180863 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:53.180821 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:53.666469 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:53.666436 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:54.051481 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:54.051389 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:54.415707 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:54.415673 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:33:54.865967 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:54.865923 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:55.476566 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:55.476534 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:56.228674 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:56.228639 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:56.940428 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:56.940373 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:57.865113 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:57.865057 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:58.523037 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:58.523004 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:59.544303 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:59.544270 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:33:59.988754 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:33:59.988727 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:00.659497 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:00.659459 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:01.466742 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:01.466653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:03.956434 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:03.956387 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:04.488231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:04.488198 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k"] Sep 16 01:34:04.489663 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:04.489633 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:04.492286 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:04.492264 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-033feb-29825370-w5j6k"] Sep 16 01:34:04.518248 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:04.518221 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p"] Sep 16 01:34:04.518462 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:04.518432 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" podUID="f59c4f2e-ce71-42f2-b374-5d9e5fa910d6" containerName="payload-pre-processing" containerID="cri-o://c6b286d2464771b69ad003340e888813bc3054e89557de647fe02a1917ac7294" gracePeriod=30 Sep 16 01:34:04.584783 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:04.584739 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:04.782553 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:04.782530 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" Sep 16 01:34:04.810802 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:04.810747 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:04.863406 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:04.863359 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:04.903716 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:04.903681 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2ddq\" (UniqueName: \"kubernetes.io/projected/f59c4f2e-ce71-42f2-b374-5d9e5fa910d6-kube-api-access-n2ddq\") pod \"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6\" (UID: \"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6\") " Sep 16 01:34:04.903875 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:04.903777 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59c4f2e-ce71-42f2-b374-5d9e5fa910d6-plugins-config-volume\") pod \"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6\" (UID: \"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6\") " Sep 16 01:34:04.904112 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:04.904091 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59c4f2e-ce71-42f2-b374-5d9e5fa910d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f59c4f2e-ce71-42f2-b374-5d9e5fa910d6" (UID: "f59c4f2e-ce71-42f2-b374-5d9e5fa910d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:34:04.905874 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:04.905852 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59c4f2e-ce71-42f2-b374-5d9e5fa910d6-kube-api-access-n2ddq" (OuterVolumeSpecName: "kube-api-access-n2ddq") pod "f59c4f2e-ce71-42f2-b374-5d9e5fa910d6" (UID: "f59c4f2e-ce71-42f2-b374-5d9e5fa910d6"). InnerVolumeSpecName "kube-api-access-n2ddq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:34:05.004690 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.004655 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59c4f2e-ce71-42f2-b374-5d9e5fa910d6-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:05.004690 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.004681 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2ddq\" (UniqueName: \"kubernetes.io/projected/f59c4f2e-ce71-42f2-b374-5d9e5fa910d6-kube-api-access-n2ddq\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:05.056759 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.056681 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:05.128998 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.128971 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:05.214438 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.214393 2558 generic.go:356] "Generic (PLEG): container finished" podID="f59c4f2e-ce71-42f2-b374-5d9e5fa910d6" containerID="c6b286d2464771b69ad003340e888813bc3054e89557de647fe02a1917ac7294" exitCode=0 Sep 16 01:34:05.214635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.214440 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" Sep 16 01:34:05.214635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.214485 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" event={"ID":"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6","Type":"ContainerDied","Data":"c6b286d2464771b69ad003340e888813bc3054e89557de647fe02a1917ac7294"} Sep 16 01:34:05.214635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.214543 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p" event={"ID":"f59c4f2e-ce71-42f2-b374-5d9e5fa910d6","Type":"ContainerDied","Data":"b56aa3dd0d5fd95f46ffbe380438f450c53a2379d66afb1c6ad1da06ebe71056"} Sep 16 01:34:05.214635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.214573 2558 scope.go:117] "RemoveContainer" containerID="c6b286d2464771b69ad003340e888813bc3054e89557de647fe02a1917ac7294" Sep 16 01:34:05.225441 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.225423 2558 scope.go:117] "RemoveContainer" containerID="c6b286d2464771b69ad003340e888813bc3054e89557de647fe02a1917ac7294" Sep 16 01:34:05.230238 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:34:05.225964 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b286d2464771b69ad003340e888813bc3054e89557de647fe02a1917ac7294\": container with ID starting with c6b286d2464771b69ad003340e888813bc3054e89557de647fe02a1917ac7294 not found: ID does not exist" containerID="c6b286d2464771b69ad003340e888813bc3054e89557de647fe02a1917ac7294" Sep 16 01:34:05.230238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.226076 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b286d2464771b69ad003340e888813bc3054e89557de647fe02a1917ac7294"} err="failed to get container status \"c6b286d2464771b69ad003340e888813bc3054e89557de647fe02a1917ac7294\": rpc error: code = NotFound desc = could not find container \"c6b286d2464771b69ad003340e888813bc3054e89557de647fe02a1917ac7294\": container with ID starting with c6b286d2464771b69ad003340e888813bc3054e89557de647fe02a1917ac7294 not found: ID does not exist" Sep 16 01:34:05.237073 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.237048 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p"] Sep 16 01:34:05.241337 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.241317 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-033feb-8bc5cf78-zpn5p"] Sep 16 01:34:05.858101 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:05.858071 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:06.296109 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:06.296071 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d52963ff-75a9-456c-b9e6-835f7310496a" path="/var/lib/kubelet/pods/d52963ff-75a9-456c-b9e6-835f7310496a/volumes" Sep 16 01:34:06.296485 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:06.296464 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f59c4f2e-ce71-42f2-b374-5d9e5fa910d6" path="/var/lib/kubelet/pods/f59c4f2e-ce71-42f2-b374-5d9e5fa910d6/volumes" Sep 16 01:34:06.662040 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:06.662004 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:07.301292 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:07.301264 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:07.938270 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:07.938235 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:07.969520 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:07.969492 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4"] Sep 16 01:34:07.969899 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:07.969867 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" podUID="3df5c422-e9d8-4c0b-b645-8276164d00b9" containerName="payload-processing" containerID="cri-o://12c9836f3177a2cb811afac3ae9ae63d6e68576d985e7fa6a1772432129759df" gracePeriod=30 Sep 16 01:34:08.037996 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.037949 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:08.126711 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.126669 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:08.184603 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.184574 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:08.238241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.238214 2558 generic.go:356] "Generic (PLEG): container finished" podID="3df5c422-e9d8-4c0b-b645-8276164d00b9" containerID="12c9836f3177a2cb811afac3ae9ae63d6e68576d985e7fa6a1772432129759df" exitCode=0 Sep 16 01:34:08.238396 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.238293 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" event={"ID":"3df5c422-e9d8-4c0b-b645-8276164d00b9","Type":"ContainerDied","Data":"12c9836f3177a2cb811afac3ae9ae63d6e68576d985e7fa6a1772432129759df"} Sep 16 01:34:08.238396 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.238359 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" event={"ID":"3df5c422-e9d8-4c0b-b645-8276164d00b9","Type":"ContainerDied","Data":"ff0676aa557a5c034aefbc72218938ed273b8785c460bfc6c1a7330a5d67a239"} Sep 16 01:34:08.238396 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.238385 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff0676aa557a5c034aefbc72218938ed273b8785c460bfc6c1a7330a5d67a239" Sep 16 01:34:08.256465 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.256435 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" Sep 16 01:34:08.260309 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.260220 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:08.338551 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.338519 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df5c422-e9d8-4c0b-b645-8276164d00b9-plugins-config-volume\") pod \"3df5c422-e9d8-4c0b-b645-8276164d00b9\" (UID: \"3df5c422-e9d8-4c0b-b645-8276164d00b9\") " Sep 16 01:34:08.338992 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.338656 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-594xz\" (UniqueName: \"kubernetes.io/projected/3df5c422-e9d8-4c0b-b645-8276164d00b9-kube-api-access-594xz\") pod \"3df5c422-e9d8-4c0b-b645-8276164d00b9\" (UID: \"3df5c422-e9d8-4c0b-b645-8276164d00b9\") " Sep 16 01:34:08.338992 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.338928 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df5c422-e9d8-4c0b-b645-8276164d00b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3df5c422-e9d8-4c0b-b645-8276164d00b9" (UID: "3df5c422-e9d8-4c0b-b645-8276164d00b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:34:08.339125 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.339032 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df5c422-e9d8-4c0b-b645-8276164d00b9-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:08.353360 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.353332 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df5c422-e9d8-4c0b-b645-8276164d00b9-kube-api-access-594xz" (OuterVolumeSpecName: "kube-api-access-594xz") pod "3df5c422-e9d8-4c0b-b645-8276164d00b9" (UID: "3df5c422-e9d8-4c0b-b645-8276164d00b9"). InnerVolumeSpecName "kube-api-access-594xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:34:08.440248 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.440207 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-594xz\" (UniqueName: \"kubernetes.io/projected/3df5c422-e9d8-4c0b-b645-8276164d00b9-kube-api-access-594xz\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:08.554898 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.554799 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:08.727275 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.727244 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:08.828640 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.828557 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:08.951946 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:08.951901 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:09.245429 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:09.245381 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4" Sep 16 01:34:09.256370 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:09.256346 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:09.274669 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:09.274639 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4"] Sep 16 01:34:09.279111 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:09.279075 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ba321c07-549495cf89-n5df4"] Sep 16 01:34:09.353826 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:09.353794 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:09.591236 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:09.591136 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:10.157335 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:10.157253 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:10.302140 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:10.302107 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3df5c422-e9d8-4c0b-b645-8276164d00b9" path="/var/lib/kubelet/pods/3df5c422-e9d8-4c0b-b645-8276164d00b9/volumes" Sep 16 01:34:10.978631 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:10.978601 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:11.097338 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:11.097249 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:11.286733 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:11.286627 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:11.722682 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:11.722652 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:11.847511 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:11.847478 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:11.948188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:11.948121 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:12.858240 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:12.858197 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:13.500694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:13.500660 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:13.601452 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:13.601422 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:13.832492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:13.832419 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:13.949345 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:13.949313 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:14.043424 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:14.043397 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:14.586424 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:14.586395 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:15.262926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:15.262881 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:15.717303 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:15.717267 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:34:15.962556 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:15.962515 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:16.564108 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:16.564062 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:21.759620 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:21.759584 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:22.282884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.282848 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn"] Sep 16 01:34:22.283956 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.283920 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:22.285667 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.285644 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a58460-29825370-9jzzn"] Sep 16 01:34:22.295679 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.295651 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5af05c9-f1cb-455a-b915-5ab99dca0f52" path="/var/lib/kubelet/pods/c5af05c9-f1cb-455a-b915-5ab99dca0f52/volumes" Sep 16 01:34:22.302754 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.302728 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t"] Sep 16 01:34:22.302994 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.302966 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" podUID="47d5ee9f-125b-4074-8ec2-ad0b92df1009" containerName="payload-processing" containerID="cri-o://6d627e850e90daeee28f1ddd21831ec6142a37f2f79e2dd98efd267f7588e4cc" gracePeriod=30 Sep 16 01:34:22.314517 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.314492 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr"] Sep 16 01:34:22.314704 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.314683 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" podUID="71958bb4-b1b5-402b-aa36-7beec61f5829" containerName="payload-pre-processing" containerID="cri-o://21d34ec313e2d6bb3eea6b42a9df3bee6f7b20ee584bfd1c138477e4e4955538" gracePeriod=30 Sep 16 01:34:22.383026 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.382998 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:22.456901 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.456840 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:22.509533 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.509505 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:22.608946 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.608922 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" Sep 16 01:34:22.612041 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.612022 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" Sep 16 01:34:22.778182 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.778125 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxd7w\" (UniqueName: \"kubernetes.io/projected/47d5ee9f-125b-4074-8ec2-ad0b92df1009-kube-api-access-bxd7w\") pod \"47d5ee9f-125b-4074-8ec2-ad0b92df1009\" (UID: \"47d5ee9f-125b-4074-8ec2-ad0b92df1009\") " Sep 16 01:34:22.778601 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.778275 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71958bb4-b1b5-402b-aa36-7beec61f5829-plugins-config-volume\") pod \"71958bb4-b1b5-402b-aa36-7beec61f5829\" (UID: \"71958bb4-b1b5-402b-aa36-7beec61f5829\") " Sep 16 01:34:22.778601 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.778298 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d5ee9f-125b-4074-8ec2-ad0b92df1009-plugins-config-volume\") pod \"47d5ee9f-125b-4074-8ec2-ad0b92df1009\" (UID: \"47d5ee9f-125b-4074-8ec2-ad0b92df1009\") " Sep 16 01:34:22.778601 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.778325 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z862p\" (UniqueName: \"kubernetes.io/projected/71958bb4-b1b5-402b-aa36-7beec61f5829-kube-api-access-z862p\") pod \"71958bb4-b1b5-402b-aa36-7beec61f5829\" (UID: \"71958bb4-b1b5-402b-aa36-7beec61f5829\") " Sep 16 01:34:22.778755 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.778701 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d5ee9f-125b-4074-8ec2-ad0b92df1009-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "47d5ee9f-125b-4074-8ec2-ad0b92df1009" (UID: "47d5ee9f-125b-4074-8ec2-ad0b92df1009"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:34:22.778755 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.778708 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71958bb4-b1b5-402b-aa36-7beec61f5829-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71958bb4-b1b5-402b-aa36-7beec61f5829" (UID: "71958bb4-b1b5-402b-aa36-7beec61f5829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:34:22.780405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.780384 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d5ee9f-125b-4074-8ec2-ad0b92df1009-kube-api-access-bxd7w" (OuterVolumeSpecName: "kube-api-access-bxd7w") pod "47d5ee9f-125b-4074-8ec2-ad0b92df1009" (UID: "47d5ee9f-125b-4074-8ec2-ad0b92df1009"). InnerVolumeSpecName "kube-api-access-bxd7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:34:22.780685 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.780667 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71958bb4-b1b5-402b-aa36-7beec61f5829-kube-api-access-z862p" (OuterVolumeSpecName: "kube-api-access-z862p") pod "71958bb4-b1b5-402b-aa36-7beec61f5829" (UID: "71958bb4-b1b5-402b-aa36-7beec61f5829"). InnerVolumeSpecName "kube-api-access-z862p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:34:22.843714 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.843653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:22.879332 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.879304 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71958bb4-b1b5-402b-aa36-7beec61f5829-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:22.879332 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.879328 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d5ee9f-125b-4074-8ec2-ad0b92df1009-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:22.879332 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.879337 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z862p\" (UniqueName: \"kubernetes.io/projected/71958bb4-b1b5-402b-aa36-7beec61f5829-kube-api-access-z862p\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:22.879512 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.879346 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxd7w\" (UniqueName: \"kubernetes.io/projected/47d5ee9f-125b-4074-8ec2-ad0b92df1009-kube-api-access-bxd7w\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:22.944477 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:22.944446 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:23.349060 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.349023 2558 generic.go:356] "Generic (PLEG): container finished" podID="71958bb4-b1b5-402b-aa36-7beec61f5829" containerID="21d34ec313e2d6bb3eea6b42a9df3bee6f7b20ee584bfd1c138477e4e4955538" exitCode=0 Sep 16 01:34:23.349291 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.349069 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" Sep 16 01:34:23.349291 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.349107 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" event={"ID":"71958bb4-b1b5-402b-aa36-7beec61f5829","Type":"ContainerDied","Data":"21d34ec313e2d6bb3eea6b42a9df3bee6f7b20ee584bfd1c138477e4e4955538"} Sep 16 01:34:23.349291 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.349169 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr" event={"ID":"71958bb4-b1b5-402b-aa36-7beec61f5829","Type":"ContainerDied","Data":"a75eff5bd3a9a576ae6950ba6e2599edd52f7b7c9aca5bd80358d18cc8cd39bb"} Sep 16 01:34:23.349291 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.349198 2558 scope.go:117] "RemoveContainer" containerID="21d34ec313e2d6bb3eea6b42a9df3bee6f7b20ee584bfd1c138477e4e4955538" Sep 16 01:34:23.350694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.350669 2558 generic.go:356] "Generic (PLEG): container finished" podID="47d5ee9f-125b-4074-8ec2-ad0b92df1009" containerID="6d627e850e90daeee28f1ddd21831ec6142a37f2f79e2dd98efd267f7588e4cc" exitCode=0 Sep 16 01:34:23.350819 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.350704 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" event={"ID":"47d5ee9f-125b-4074-8ec2-ad0b92df1009","Type":"ContainerDied","Data":"6d627e850e90daeee28f1ddd21831ec6142a37f2f79e2dd98efd267f7588e4cc"} Sep 16 01:34:23.350819 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.350710 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" Sep 16 01:34:23.350819 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.350744 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t" event={"ID":"47d5ee9f-125b-4074-8ec2-ad0b92df1009","Type":"ContainerDied","Data":"f71b6983524c376f849336f8997f6aa0857359487473a8f313d8687dbd34d56d"} Sep 16 01:34:23.360126 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.360107 2558 scope.go:117] "RemoveContainer" containerID="21d34ec313e2d6bb3eea6b42a9df3bee6f7b20ee584bfd1c138477e4e4955538" Sep 16 01:34:23.360585 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:34:23.360447 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d34ec313e2d6bb3eea6b42a9df3bee6f7b20ee584bfd1c138477e4e4955538\": container with ID starting with 21d34ec313e2d6bb3eea6b42a9df3bee6f7b20ee584bfd1c138477e4e4955538 not found: ID does not exist" containerID="21d34ec313e2d6bb3eea6b42a9df3bee6f7b20ee584bfd1c138477e4e4955538" Sep 16 01:34:23.360585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.360483 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d34ec313e2d6bb3eea6b42a9df3bee6f7b20ee584bfd1c138477e4e4955538"} err="failed to get container status \"21d34ec313e2d6bb3eea6b42a9df3bee6f7b20ee584bfd1c138477e4e4955538\": rpc error: code = NotFound desc = could not find container \"21d34ec313e2d6bb3eea6b42a9df3bee6f7b20ee584bfd1c138477e4e4955538\": container with ID starting with 21d34ec313e2d6bb3eea6b42a9df3bee6f7b20ee584bfd1c138477e4e4955538 not found: ID does not exist" Sep 16 01:34:23.360585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.360507 2558 scope.go:117] "RemoveContainer" containerID="6d627e850e90daeee28f1ddd21831ec6142a37f2f79e2dd98efd267f7588e4cc" Sep 16 01:34:23.369483 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.369466 2558 scope.go:117] "RemoveContainer" containerID="6d627e850e90daeee28f1ddd21831ec6142a37f2f79e2dd98efd267f7588e4cc" Sep 16 01:34:23.369789 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:34:23.369771 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d627e850e90daeee28f1ddd21831ec6142a37f2f79e2dd98efd267f7588e4cc\": container with ID starting with 6d627e850e90daeee28f1ddd21831ec6142a37f2f79e2dd98efd267f7588e4cc not found: ID does not exist" containerID="6d627e850e90daeee28f1ddd21831ec6142a37f2f79e2dd98efd267f7588e4cc" Sep 16 01:34:23.369889 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.369795 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d627e850e90daeee28f1ddd21831ec6142a37f2f79e2dd98efd267f7588e4cc"} err="failed to get container status \"6d627e850e90daeee28f1ddd21831ec6142a37f2f79e2dd98efd267f7588e4cc\": rpc error: code = NotFound desc = could not find container \"6d627e850e90daeee28f1ddd21831ec6142a37f2f79e2dd98efd267f7588e4cc\": container with ID starting with 6d627e850e90daeee28f1ddd21831ec6142a37f2f79e2dd98efd267f7588e4cc not found: ID does not exist" Sep 16 01:34:23.371711 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.371687 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr"] Sep 16 01:34:23.374380 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.374358 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a58460-ff8fb5f78-f6msr"] Sep 16 01:34:23.382695 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.382674 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t"] Sep 16 01:34:23.387888 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.387869 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a58460-6b9597c679-kct4t"] Sep 16 01:34:23.577392 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:23.577359 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:24.295670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:24.295642 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47d5ee9f-125b-4074-8ec2-ad0b92df1009" path="/var/lib/kubelet/pods/47d5ee9f-125b-4074-8ec2-ad0b92df1009/volumes" Sep 16 01:34:24.296028 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:24.295993 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71958bb4-b1b5-402b-aa36-7beec61f5829" path="/var/lib/kubelet/pods/71958bb4-b1b5-402b-aa36-7beec61f5829/volumes" Sep 16 01:34:24.654008 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:24.653974 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:25.345398 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:25.345361 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:26.173195 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.173147 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:26.521058 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.520984 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk"] Sep 16 01:34:26.521430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.521400 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71958bb4-b1b5-402b-aa36-7beec61f5829" containerName="payload-pre-processing" Sep 16 01:34:26.521430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.521413 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="71958bb4-b1b5-402b-aa36-7beec61f5829" containerName="payload-pre-processing" Sep 16 01:34:26.521430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.521429 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d5ee9f-125b-4074-8ec2-ad0b92df1009" containerName="payload-processing" Sep 16 01:34:26.521542 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.521434 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d5ee9f-125b-4074-8ec2-ad0b92df1009" containerName="payload-processing" Sep 16 01:34:26.521542 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.521446 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df5c422-e9d8-4c0b-b645-8276164d00b9" containerName="payload-processing" Sep 16 01:34:26.521542 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.521452 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df5c422-e9d8-4c0b-b645-8276164d00b9" containerName="payload-processing" Sep 16 01:34:26.521542 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.521467 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f59c4f2e-ce71-42f2-b374-5d9e5fa910d6" containerName="payload-pre-processing" Sep 16 01:34:26.521542 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.521473 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f59c4f2e-ce71-42f2-b374-5d9e5fa910d6" containerName="payload-pre-processing" Sep 16 01:34:26.521694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.521559 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f59c4f2e-ce71-42f2-b374-5d9e5fa910d6" containerName="payload-pre-processing" Sep 16 01:34:26.521694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.521568 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="47d5ee9f-125b-4074-8ec2-ad0b92df1009" containerName="payload-processing" Sep 16 01:34:26.521694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.521576 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3df5c422-e9d8-4c0b-b645-8276164d00b9" containerName="payload-processing" Sep 16 01:34:26.521694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.521583 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="71958bb4-b1b5-402b-aa36-7beec61f5829" containerName="payload-pre-processing" Sep 16 01:34:26.526269 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.526245 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk" Sep 16 01:34:26.528704 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.528546 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 01:34:26.528888 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.528814 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4ljjd\"" Sep 16 01:34:26.531344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.531320 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk"] Sep 16 01:34:26.614019 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.613984 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7795868a-a8de-4827-9d3d-e10486e62ee2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk\" (UID: \"7795868a-a8de-4827-9d3d-e10486e62ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk" Sep 16 01:34:26.714658 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.714628 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7795868a-a8de-4827-9d3d-e10486e62ee2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk\" (UID: \"7795868a-a8de-4827-9d3d-e10486e62ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk" Sep 16 01:34:26.715316 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.715295 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7795868a-a8de-4827-9d3d-e10486e62ee2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk\" (UID: \"7795868a-a8de-4827-9d3d-e10486e62ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk" Sep 16 01:34:26.839300 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:26.839229 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk" Sep 16 01:34:27.065519 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:27.065484 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:27.111717 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:27.111695 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk"] Sep 16 01:34:27.113807 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:34:27.113772 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7795868a_a8de_4827_9d3d_e10486e62ee2.slice/crio-719e2ebbd1d875904f61e006c1052069ff7bc30ec27e3e6798db6c084e31822b WatchSource:0}: Error finding container 719e2ebbd1d875904f61e006c1052069ff7bc30ec27e3e6798db6c084e31822b: Status 404 returned error can't find the container with id 719e2ebbd1d875904f61e006c1052069ff7bc30ec27e3e6798db6c084e31822b Sep 16 01:34:27.388141 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:27.388114 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk" event={"ID":"7795868a-a8de-4827-9d3d-e10486e62ee2","Type":"ContainerStarted","Data":"7c46e49aec43e51ed028a085fee494de1ee940e1164d9bb4b4020984236fa606"} Sep 16 01:34:27.388278 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:27.388183 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk" event={"ID":"7795868a-a8de-4827-9d3d-e10486e62ee2","Type":"ContainerStarted","Data":"719e2ebbd1d875904f61e006c1052069ff7bc30ec27e3e6798db6c084e31822b"} Sep 16 01:34:27.401391 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:27.401340 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk" podStartSLOduration=1.401321775 podStartE2EDuration="1.401321775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:34:27.400433908 +0000 UTC m=+1097.652856497" watchObservedRunningTime="2026-09-16 01:34:27.401321775 +0000 UTC m=+1097.653744364" Sep 16 01:34:27.662119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:27.662033 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:28.396401 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:28.396367 2558 generic.go:356] "Generic (PLEG): container finished" podID="7795868a-a8de-4827-9d3d-e10486e62ee2" containerID="7c46e49aec43e51ed028a085fee494de1ee940e1164d9bb4b4020984236fa606" exitCode=0 Sep 16 01:34:28.396585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:28.396435 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk" event={"ID":"7795868a-a8de-4827-9d3d-e10486e62ee2","Type":"ContainerDied","Data":"7c46e49aec43e51ed028a085fee494de1ee940e1164d9bb4b4020984236fa606"} Sep 16 01:34:28.494141 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:28.494110 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:28.774680 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:28.774589 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:28.920420 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:28.920382 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:29.105293 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.105215 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q"] Sep 16 01:34:29.105459 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.105434 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" podUID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerName="istio-proxy" containerID="cri-o://324927dfa6df7beb692965e14fa025ef27a2e11a6c0d3980c6180193fe32c71b" gracePeriod=30 Sep 16 01:34:29.122885 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.122851 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:29.529702 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.529677 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk" Sep 16 01:34:29.645213 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.645186 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7795868a-a8de-4827-9d3d-e10486e62ee2-maas-api-service-ca\") pod \"7795868a-a8de-4827-9d3d-e10486e62ee2\" (UID: \"7795868a-a8de-4827-9d3d-e10486e62ee2\") " Sep 16 01:34:29.645556 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.645531 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7795868a-a8de-4827-9d3d-e10486e62ee2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7795868a-a8de-4827-9d3d-e10486e62ee2" (UID: "7795868a-a8de-4827-9d3d-e10486e62ee2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:34:29.701867 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.701788 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:29.746892 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.746865 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7795868a-a8de-4827-9d3d-e10486e62ee2-maas-api-service-ca\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:29.981108 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.981027 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr"] Sep 16 01:34:29.981728 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.981712 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7795868a-a8de-4827-9d3d-e10486e62ee2" containerName="revoke-keys" Sep 16 01:34:29.981769 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.981732 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7795868a-a8de-4827-9d3d-e10486e62ee2" containerName="revoke-keys" Sep 16 01:34:29.981835 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.981824 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7795868a-a8de-4827-9d3d-e10486e62ee2" containerName="revoke-keys" Sep 16 01:34:29.985346 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.985323 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:29.987392 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.987375 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-cbb499-openshift-default-dockercfg-vk822\"" Sep 16 01:34:29.995940 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.995913 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr"] Sep 16 01:34:29.996321 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:29.996287 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:30.150631 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.150592 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istiod-ca-cert\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.150798 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.150640 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-credential-socket\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.150798 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.150659 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-token\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.150798 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.150702 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-podinfo\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.150798 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.150725 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-workload-certs\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.150942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.150815 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-workload-socket\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.150942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.150864 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-748l7\" (UniqueName: \"kubernetes.io/projected/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-kube-api-access-748l7\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.150942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.150914 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-envoy\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.151040 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.150941 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-data\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.249590 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.249522 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:30.251748 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.251728 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-workload-socket\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.251807 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.251763 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-748l7\" (UniqueName: \"kubernetes.io/projected/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-kube-api-access-748l7\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.251807 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.251790 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-envoy\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.251873 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.251814 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-data\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.251873 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.251845 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istiod-ca-cert\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.251959 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.251882 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-credential-socket\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.251959 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.251909 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-token\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.251959 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.251935 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-podinfo\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.252105 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.251967 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-workload-certs\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.252318 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.252285 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-workload-socket\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.252452 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.252359 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-data\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.252452 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.252390 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-workload-certs\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.252535 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.252452 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-credential-socket\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.252587 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.252554 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istiod-ca-cert\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.254270 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.254250 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-envoy\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.254846 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.254825 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-podinfo\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.259097 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.259067 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-token\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.259502 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.259480 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-748l7\" (UniqueName: \"kubernetes.io/projected/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-kube-api-access-748l7\") pod \"e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.296938 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.296909 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:30.409554 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.409523 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk" event={"ID":"7795868a-a8de-4827-9d3d-e10486e62ee2","Type":"ContainerDied","Data":"719e2ebbd1d875904f61e006c1052069ff7bc30ec27e3e6798db6c084e31822b"} Sep 16 01:34:30.409554 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.409549 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk" Sep 16 01:34:30.409742 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.409573 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="719e2ebbd1d875904f61e006c1052069ff7bc30ec27e3e6798db6c084e31822b" Sep 16 01:34:30.581430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.581403 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr"] Sep 16 01:34:30.582585 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:34:30.582553 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc26edd_b9e5_41b5_a015_e2f9f63e2586.slice/crio-094e6f0ee9a0439284fb346ae4b984f7240720bad37aae960430f4f7fb6b66e8 WatchSource:0}: Error finding container 094e6f0ee9a0439284fb346ae4b984f7240720bad37aae960430f4f7fb6b66e8: Status 404 returned error can't find the container with id 094e6f0ee9a0439284fb346ae4b984f7240720bad37aae960430f4f7fb6b66e8 Sep 16 01:34:30.584938 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.584903 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:34:30.585030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.584967 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:34:30.585030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:30.585019 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:34:31.416801 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:31.416761 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" event={"ID":"bbc26edd-b9e5-41b5-a015-e2f9f63e2586","Type":"ContainerStarted","Data":"265c4193c1e521622e3ad93ff62029c4b8f8f6cd81de00980804cbbbf0c9c110"} Sep 16 01:34:31.417257 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:31.416816 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" event={"ID":"bbc26edd-b9e5-41b5-a015-e2f9f63e2586","Type":"ContainerStarted","Data":"094e6f0ee9a0439284fb346ae4b984f7240720bad37aae960430f4f7fb6b66e8"} Sep 16 01:34:31.435688 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:31.435643 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" podStartSLOduration=2.435626727 podStartE2EDuration="2.435626727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:34:31.432805924 +0000 UTC m=+1101.685228514" watchObservedRunningTime="2026-09-16 01:34:31.435626727 +0000 UTC m=+1101.688049316" Sep 16 01:34:32.297454 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:32.297427 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:32.302559 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:32.302534 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:32.424125 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:32.424097 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:32.424969 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:32.424946 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:32.576219 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:32.576077 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:33.304370 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:33.304338 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:33.398683 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:33.398651 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:33.499224 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:33.499193 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:34.274339 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.274310 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:34.368559 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.368494 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:34:34.400935 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.400912 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-podinfo\") pod \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " Sep 16 01:34:34.401069 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.400947 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-token\") pod \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " Sep 16 01:34:34.401069 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.400970 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-data\") pod \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " Sep 16 01:34:34.401069 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401023 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-credential-socket\") pod \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " Sep 16 01:34:34.401249 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401070 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-workload-socket\") pod \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " Sep 16 01:34:34.401249 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401137 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxk2n\" (UniqueName: \"kubernetes.io/projected/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-kube-api-access-bxk2n\") pod \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " Sep 16 01:34:34.401249 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401179 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istiod-ca-cert\") pod \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " Sep 16 01:34:34.401249 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401209 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-envoy\") pod \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " Sep 16 01:34:34.401249 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401237 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-workload-certs\") pod \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\" (UID: \"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea\") " Sep 16 01:34:34.401531 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401374 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" (UID: "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:34:34.401531 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401397 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" (UID: "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:34:34.401643 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401531 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" (UID: "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:34:34.401643 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401627 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" (UID: "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:34:34.401758 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401725 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istiod-ca-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:34.401758 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401744 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-data\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:34.401858 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401757 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-credential-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:34.401858 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401772 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-workload-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:34.402030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.401977 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" (UID: "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:34:34.403618 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.403587 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" (UID: "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:34:34.403729 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.403675 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-kube-api-access-bxk2n" (OuterVolumeSpecName: "kube-api-access-bxk2n") pod "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" (UID: "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea"). InnerVolumeSpecName "kube-api-access-bxk2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:34:34.403926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.403770 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" (UID: "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:34:34.403926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.403784 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" (UID: "2d9db5a2-7b97-43b2-8fa1-116d8cf2abea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:34:34.438402 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.438368 2558 generic.go:356] "Generic (PLEG): container finished" podID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerID="324927dfa6df7beb692965e14fa025ef27a2e11a6c0d3980c6180193fe32c71b" exitCode=0 Sep 16 01:34:34.438509 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.438427 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" Sep 16 01:34:34.438509 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.438462 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" event={"ID":"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea","Type":"ContainerDied","Data":"324927dfa6df7beb692965e14fa025ef27a2e11a6c0d3980c6180193fe32c71b"} Sep 16 01:34:34.438596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.438517 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q" event={"ID":"2d9db5a2-7b97-43b2-8fa1-116d8cf2abea","Type":"ContainerDied","Data":"f8e059b5e214fbade6d6fd9462c16b592b10ce49284db42c86796f0ee4109412"} Sep 16 01:34:34.438596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.438546 2558 scope.go:117] "RemoveContainer" containerID="324927dfa6df7beb692965e14fa025ef27a2e11a6c0d3980c6180193fe32c71b" Sep 16 01:34:34.449467 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.449450 2558 scope.go:117] "RemoveContainer" containerID="9652ad4f0ed457f913ce019d0041833b3f6776760f4a1c18da683ae4210cd937" Sep 16 01:34:34.458799 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.458779 2558 scope.go:117] "RemoveContainer" containerID="324927dfa6df7beb692965e14fa025ef27a2e11a6c0d3980c6180193fe32c71b" Sep 16 01:34:34.459072 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:34:34.459051 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324927dfa6df7beb692965e14fa025ef27a2e11a6c0d3980c6180193fe32c71b\": container with ID starting with 324927dfa6df7beb692965e14fa025ef27a2e11a6c0d3980c6180193fe32c71b not found: ID does not exist" containerID="324927dfa6df7beb692965e14fa025ef27a2e11a6c0d3980c6180193fe32c71b" Sep 16 01:34:34.459149 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.459080 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324927dfa6df7beb692965e14fa025ef27a2e11a6c0d3980c6180193fe32c71b"} err="failed to get container status \"324927dfa6df7beb692965e14fa025ef27a2e11a6c0d3980c6180193fe32c71b\": rpc error: code = NotFound desc = could not find container \"324927dfa6df7beb692965e14fa025ef27a2e11a6c0d3980c6180193fe32c71b\": container with ID starting with 324927dfa6df7beb692965e14fa025ef27a2e11a6c0d3980c6180193fe32c71b not found: ID does not exist" Sep 16 01:34:34.459149 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.459096 2558 scope.go:117] "RemoveContainer" containerID="9652ad4f0ed457f913ce019d0041833b3f6776760f4a1c18da683ae4210cd937" Sep 16 01:34:34.459930 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:34:34.459904 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9652ad4f0ed457f913ce019d0041833b3f6776760f4a1c18da683ae4210cd937\": container with ID starting with 9652ad4f0ed457f913ce019d0041833b3f6776760f4a1c18da683ae4210cd937 not found: ID does not exist" containerID="9652ad4f0ed457f913ce019d0041833b3f6776760f4a1c18da683ae4210cd937" Sep 16 01:34:34.460013 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.459938 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9652ad4f0ed457f913ce019d0041833b3f6776760f4a1c18da683ae4210cd937"} err="failed to get container status \"9652ad4f0ed457f913ce019d0041833b3f6776760f4a1c18da683ae4210cd937\": rpc error: code = NotFound desc = could not find container \"9652ad4f0ed457f913ce019d0041833b3f6776760f4a1c18da683ae4210cd937\": container with ID starting with 9652ad4f0ed457f913ce019d0041833b3f6776760f4a1c18da683ae4210cd937 not found: ID does not exist" Sep 16 01:34:34.461640 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.461615 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q"] Sep 16 01:34:34.465406 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.465376 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a58460-openshift-default-7fb6cb84f6-qqm2q"] Sep 16 01:34:34.502819 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.502793 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxk2n\" (UniqueName: \"kubernetes.io/projected/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-kube-api-access-bxk2n\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:34.502819 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.502815 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-envoy\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:34.503239 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.502826 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-workload-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:34.503239 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.502834 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-podinfo\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:34.503239 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:34.502842 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea-istio-token\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:34:35.059745 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:35.059709 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:35.217961 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:35.217930 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:34:35.877837 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:35.877806 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:35.977611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:35.977573 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:36.294889 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.294807 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" path="/var/lib/kubelet/pods/2d9db5a2-7b97-43b2-8fa1-116d8cf2abea/volumes" Sep 16 01:34:36.415974 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.415946 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:36.507378 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.507346 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:36.511888 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.511864 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z"] Sep 16 01:34:36.512300 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.512286 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerName="istio-proxy" Sep 16 01:34:36.512300 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.512301 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerName="istio-proxy" Sep 16 01:34:36.512443 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.512373 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerName="istio-proxy" Sep 16 01:34:36.515703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.515682 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" Sep 16 01:34:36.517779 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.517760 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-cbb499\"" Sep 16 01:34:36.517878 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.517810 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-cbb499-dockercfg-62ptf\"" Sep 16 01:34:36.525905 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.525879 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z"] Sep 16 01:34:36.553209 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.553102 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7"] Sep 16 01:34:36.553535 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.553518 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerName="istio-proxy" Sep 16 01:34:36.553535 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.553533 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerName="istio-proxy" Sep 16 01:34:36.554632 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.554605 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9db5a2-7b97-43b2-8fa1-116d8cf2abea" containerName="istio-proxy" Sep 16 01:34:36.563991 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.563966 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" Sep 16 01:34:36.568018 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.567993 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7"] Sep 16 01:34:36.622650 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.622611 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e572b8a-1c4c-497f-8b11-195b92f33c0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7\" (UID: \"8e572b8a-1c4c-497f-8b11-195b92f33c0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" Sep 16 01:34:36.622845 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.622687 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwl8\" (UniqueName: \"kubernetes.io/projected/5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8-kube-api-access-cfwl8\") pod \"payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z\" (UID: \"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" Sep 16 01:34:36.622845 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.622783 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z\" (UID: \"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" Sep 16 01:34:36.622994 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.622848 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcz4r\" (UniqueName: \"kubernetes.io/projected/8e572b8a-1c4c-497f-8b11-195b92f33c0c-kube-api-access-zcz4r\") pod \"payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7\" (UID: \"8e572b8a-1c4c-497f-8b11-195b92f33c0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" Sep 16 01:34:36.724085 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.724054 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e572b8a-1c4c-497f-8b11-195b92f33c0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7\" (UID: \"8e572b8a-1c4c-497f-8b11-195b92f33c0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" Sep 16 01:34:36.724250 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.724114 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwl8\" (UniqueName: \"kubernetes.io/projected/5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8-kube-api-access-cfwl8\") pod \"payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z\" (UID: \"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" Sep 16 01:34:36.724250 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.724189 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z\" (UID: \"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" Sep 16 01:34:36.724250 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.724233 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcz4r\" (UniqueName: \"kubernetes.io/projected/8e572b8a-1c4c-497f-8b11-195b92f33c0c-kube-api-access-zcz4r\") pod \"payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7\" (UID: \"8e572b8a-1c4c-497f-8b11-195b92f33c0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" Sep 16 01:34:36.724989 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.724964 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e572b8a-1c4c-497f-8b11-195b92f33c0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7\" (UID: \"8e572b8a-1c4c-497f-8b11-195b92f33c0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" Sep 16 01:34:36.725036 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.724994 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z\" (UID: \"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" Sep 16 01:34:36.731742 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.731717 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcz4r\" (UniqueName: \"kubernetes.io/projected/8e572b8a-1c4c-497f-8b11-195b92f33c0c-kube-api-access-zcz4r\") pod \"payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7\" (UID: \"8e572b8a-1c4c-497f-8b11-195b92f33c0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" Sep 16 01:34:36.731872 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.731855 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwl8\" (UniqueName: \"kubernetes.io/projected/5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8-kube-api-access-cfwl8\") pod \"payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z\" (UID: \"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" Sep 16 01:34:36.828465 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.828404 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" Sep 16 01:34:36.877683 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.877648 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" Sep 16 01:34:36.972421 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:36.972392 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:37.101801 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.101778 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z"] Sep 16 01:34:37.103330 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:34:37.103304 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5867dbc8_1438_4f0a_9d6d_a2c1d8fd1bb8.slice/crio-96e3a3838f560c273a341a9ded3a3125e0dcd6e5b6b21db997f238fdd109bb3f WatchSource:0}: Error finding container 96e3a3838f560c273a341a9ded3a3125e0dcd6e5b6b21db997f238fdd109bb3f: Status 404 returned error can't find the container with id 96e3a3838f560c273a341a9ded3a3125e0dcd6e5b6b21db997f238fdd109bb3f Sep 16 01:34:37.170051 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.170022 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7"] Sep 16 01:34:37.170316 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:34:37.170289 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e572b8a_1c4c_497f_8b11_195b92f33c0c.slice/crio-1d4086aeaf1d4bceab3086b41cac6114296638c5a361d5d15f4fc1fd47185272 WatchSource:0}: Error finding container 1d4086aeaf1d4bceab3086b41cac6114296638c5a361d5d15f4fc1fd47185272: Status 404 returned error can't find the container with id 1d4086aeaf1d4bceab3086b41cac6114296638c5a361d5d15f4fc1fd47185272 Sep 16 01:34:37.460686 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.460440 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" event={"ID":"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8","Type":"ContainerStarted","Data":"9494fc825f17f4b8ca8043e04604ed375045f57f5e6d8dfde5f6f26f5c61f4a2"} Sep 16 01:34:37.460686 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.460525 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" Sep 16 01:34:37.460686 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.460552 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" event={"ID":"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8","Type":"ContainerStarted","Data":"96e3a3838f560c273a341a9ded3a3125e0dcd6e5b6b21db997f238fdd109bb3f"} Sep 16 01:34:37.462730 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.462699 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" event={"ID":"8e572b8a-1c4c-497f-8b11-195b92f33c0c","Type":"ContainerStarted","Data":"4ec6e73fe87ac8f74a78118858977d04246b8e77fd060d5030138dcd582d0bf0"} Sep 16 01:34:37.462877 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.462752 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" event={"ID":"8e572b8a-1c4c-497f-8b11-195b92f33c0c","Type":"ContainerStarted","Data":"1d4086aeaf1d4bceab3086b41cac6114296638c5a361d5d15f4fc1fd47185272"} Sep 16 01:34:37.462877 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.462792 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" Sep 16 01:34:37.477552 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.477506 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" podStartSLOduration=1.4774907480000001 podStartE2EDuration="1.477490748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:34:37.47548856 +0000 UTC m=+1107.727911150" watchObservedRunningTime="2026-09-16 01:34:37.477490748 +0000 UTC m=+1107.729913345" Sep 16 01:34:37.494486 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.494438 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" podStartSLOduration=1.494421677 podStartE2EDuration="1.494421677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:34:37.490889364 +0000 UTC m=+1107.743311954" watchObservedRunningTime="2026-09-16 01:34:37.494421677 +0000 UTC m=+1107.746844265" Sep 16 01:34:37.536865 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.536832 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:37.611247 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.611207 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:37.666618 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.666577 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:37.718647 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.718573 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk"] Sep 16 01:34:37.721240 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.721212 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3f6bf6aa-c7mvk"] Sep 16 01:34:37.722088 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.722067 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:37.787151 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.785111 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:37.868179 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:37.868010 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:38.294994 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:38.294962 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7795868a-a8de-4827-9d3d-e10486e62ee2" path="/var/lib/kubelet/pods/7795868a-a8de-4827-9d3d-e10486e62ee2/volumes" Sep 16 01:34:38.353259 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:38.353229 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:38.669524 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:38.669494 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:38.760430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:38.760327 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:38.816188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:38.816115 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:39.473449 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:39.473418 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:40.259077 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:40.259042 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:40.805835 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:40.805806 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:40.895413 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:40.895383 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:41.183724 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:41.183690 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:41.858122 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:41.858091 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:42.655073 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:42.655037 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:42.754695 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:42.754661 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:43.099307 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:43.099228 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:43.196509 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:43.196476 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:43.573504 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:43.573477 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:44.243591 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:44.243553 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:44.854379 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:44.854349 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:48.472985 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:48.472940 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" Sep 16 01:34:48.473988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:48.473970 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" Sep 16 01:34:55.582723 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:55.582648 2558 generic.go:356] "Generic (PLEG): container finished" podID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerID="265c4193c1e521622e3ad93ff62029c4b8f8f6cd81de00980804cbbbf0c9c110" exitCode=0 Sep 16 01:34:55.583106 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:55.582720 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" event={"ID":"bbc26edd-b9e5-41b5-a015-e2f9f63e2586","Type":"ContainerDied","Data":"265c4193c1e521622e3ad93ff62029c4b8f8f6cd81de00980804cbbbf0c9c110"} Sep 16 01:34:55.583106 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:55.583096 2558 scope.go:117] "RemoveContainer" containerID="265c4193c1e521622e3ad93ff62029c4b8f8f6cd81de00980804cbbbf0c9c110" Sep 16 01:34:55.584208 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:55.584153 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:34:55.584318 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:55.584251 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:34:55.584318 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:55.584283 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:34:56.595628 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:56.595597 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" event={"ID":"bbc26edd-b9e5-41b5-a015-e2f9f63e2586","Type":"ContainerStarted","Data":"bbd68e82209079453143de631e622817eadec545f17a3e8f92aef8abffbc01a8"} Sep 16 01:34:56.896965 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:56.896930 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:57.297274 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:57.297198 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:34:57.297414 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:57.297379 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 16 01:34:57.297459 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:57.297424 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" podUID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 16 01:34:58.298037 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:58.298007 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 16 01:34:58.298418 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:58.298062 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" podUID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 16 01:34:58.459245 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:58.459212 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:58.561285 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:58.561192 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:58.734635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:58.734594 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:58.985583 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:58.985541 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:59.094011 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:59.093843 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:59.297667 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:59.297580 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 16 01:34:59.297667 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:59.297639 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" podUID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 16 01:34:59.349620 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:59.349583 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:34:59.648417 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:34:59.648382 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:00.297815 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:00.297776 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 16 01:35:00.298015 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:00.297836 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" podUID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 16 01:35:00.455119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:00.455090 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:00.824122 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:00.824091 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:01.298031 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:01.297990 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 16 01:35:01.298259 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:01.298053 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" podUID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 16 01:35:01.376185 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:01.376131 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:02.302806 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:02.302779 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:35:02.303224 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:02.303037 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:35:02.303810 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:02.303791 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:35:02.652391 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:02.652363 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:02.721547 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:02.721518 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:35:02.753090 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:02.753054 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:03.667405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:03.667376 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:04.383834 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:04.383763 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:04.725224 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:04.725124 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:05.106769 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.106673 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:05.209674 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.209642 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:05.308097 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.308058 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:05.555522 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.555491 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:05.693074 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.693044 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd"] Sep 16 01:35:05.697861 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.697840 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" Sep 16 01:35:05.700108 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.700073 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c0ec73-dockercfg-ljzvm\"" Sep 16 01:35:05.700303 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.700083 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c0ec73\"" Sep 16 01:35:05.704670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.704646 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd"] Sep 16 01:35:05.704848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.704665 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:05.723126 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.723101 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch6fd\" (UniqueName: \"kubernetes.io/projected/9774ea02-8641-4166-9c73-4551543fe5f3-kube-api-access-ch6fd\") pod \"payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd\" (UID: \"9774ea02-8641-4166-9c73-4551543fe5f3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" Sep 16 01:35:05.723236 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.723138 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9774ea02-8641-4166-9c73-4551543fe5f3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd\" (UID: \"9774ea02-8641-4166-9c73-4551543fe5f3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" Sep 16 01:35:05.824536 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.824448 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9774ea02-8641-4166-9c73-4551543fe5f3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd\" (UID: \"9774ea02-8641-4166-9c73-4551543fe5f3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" Sep 16 01:35:05.824908 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.824585 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch6fd\" (UniqueName: \"kubernetes.io/projected/9774ea02-8641-4166-9c73-4551543fe5f3-kube-api-access-ch6fd\") pod \"payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd\" (UID: \"9774ea02-8641-4166-9c73-4551543fe5f3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" Sep 16 01:35:05.825209 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.825189 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9774ea02-8641-4166-9c73-4551543fe5f3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd\" (UID: \"9774ea02-8641-4166-9c73-4551543fe5f3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" Sep 16 01:35:05.832118 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:05.832097 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch6fd\" (UniqueName: \"kubernetes.io/projected/9774ea02-8641-4166-9c73-4551543fe5f3-kube-api-access-ch6fd\") pod \"payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd\" (UID: \"9774ea02-8641-4166-9c73-4551543fe5f3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" Sep 16 01:35:06.009844 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:06.009818 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" Sep 16 01:35:06.065400 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:06.065370 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:06.287236 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:06.287206 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd"] Sep 16 01:35:06.288073 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:35:06.288048 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9774ea02_8641_4166_9c73_4551543fe5f3.slice/crio-c91fd7bb71284b70a9279214b06495812874ae445ff6e0cc657f248677b04680 WatchSource:0}: Error finding container c91fd7bb71284b70a9279214b06495812874ae445ff6e0cc657f248677b04680: Status 404 returned error can't find the container with id c91fd7bb71284b70a9279214b06495812874ae445ff6e0cc657f248677b04680 Sep 16 01:35:06.663387 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:06.663343 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" event={"ID":"9774ea02-8641-4166-9c73-4551543fe5f3","Type":"ContainerStarted","Data":"c67a17a25be544d77a71bb2eb4e950f1e22793ec26be3689b50bbc58109f8987"} Sep 16 01:35:06.663595 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:06.663411 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" event={"ID":"9774ea02-8641-4166-9c73-4551543fe5f3","Type":"ContainerStarted","Data":"c91fd7bb71284b70a9279214b06495812874ae445ff6e0cc657f248677b04680"} Sep 16 01:35:06.663685 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:06.663655 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" Sep 16 01:35:06.681361 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:06.681304 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" podStartSLOduration=1.681283685 podStartE2EDuration="1.681283685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:35:06.678654868 +0000 UTC m=+1136.931077458" watchObservedRunningTime="2026-09-16 01:35:06.681283685 +0000 UTC m=+1136.933706275" Sep 16 01:35:06.765564 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:06.765531 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:07.567316 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:07.567283 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:11.774710 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:11.774676 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:12.293252 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:12.293215 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:12.392840 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:12.392806 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:12.449455 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:12.449422 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:12.503133 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:12.503104 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:12.880049 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:12.880018 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:13.552318 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:13.552279 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:13.825132 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:13.825046 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:14.208898 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:14.208867 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:14.309323 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:14.309293 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:14.615701 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:14.615667 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:14.990131 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:14.990039 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:15.095333 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:15.095284 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:16.270908 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:16.270862 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:17.004086 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:17.004053 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:17.349670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:17.349588 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:17.429885 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:17.429849 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:17.673528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:17.673502 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" Sep 16 01:35:18.236572 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:18.236541 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:18.951649 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:18.951614 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:19.766789 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:19.766750 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:19.819769 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:19.819737 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:19.920295 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:19.920265 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:20.205253 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:20.205211 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:20.293055 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:20.293000 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:20.562617 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:20.562531 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:21.350632 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:21.350597 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:22.057615 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:22.057561 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:24.459251 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:24.459220 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:25.959206 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:25.959153 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:30.815508 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:30.815471 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:35:40.659123 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:40.659036 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:41.770924 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:41.770859 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:42.587438 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:42.587405 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:43.344422 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:43.344386 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:43.963540 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:43.963507 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:44.775920 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:44.775887 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:45.479741 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:45.479701 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:46.154643 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:46.154601 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:48.642968 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:48.642922 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:49.187001 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:49.186952 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:49.286523 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:49.286474 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:49.363688 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:49.363655 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:49.421951 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:49.421921 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:49.512452 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:49.512371 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:49.771736 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:49.771646 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:50.252682 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:50.252653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:50.353010 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:50.352982 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:50.450494 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:50.450458 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:50.773522 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:50.773492 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:51.564703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:51.564663 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:52.052198 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:52.052141 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:52.357817 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:52.357733 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:56.022770 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:56.022732 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:57.573065 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:57.573034 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:57.639694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:57.639672 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:57.723420 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:57.723370 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:57.821642 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:57.821613 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:57.921711 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:57.921675 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:58.309344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:58.309266 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:35:58.327534 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:35:58.327510 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:36:10.298880 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:10.298851 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:36:10.301989 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:10.301964 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:36:10.362048 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:10.362021 2558 scope.go:117] "RemoveContainer" containerID="b2ee7065cfa95f29b37bcc42dc2bf7dddd6213d0469c8ac5a1d21e2026bfe68c" Sep 16 01:36:10.372548 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:10.372529 2558 scope.go:117] "RemoveContainer" containerID="8d9ec735819be6c52cf9bda6912dacc55ae6726829c8052fc2bff6d5181c9ff3" Sep 16 01:36:16.331965 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:16.331935 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:36:23.831032 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:23.830995 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:36:24.306541 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:24.306503 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:36:30.251583 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:30.251545 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:36:30.307995 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:30.307960 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:36:30.385107 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:30.385076 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:36:30.487524 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:30.487496 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:36:46.380542 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:46.380500 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:36:49.111627 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:49.111594 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:36:50.316787 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:50.316748 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:36:52.152539 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:52.152506 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:36:52.260121 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:52.260067 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:36:52.354010 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:36:52.353979 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:00.469614 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:00.469575 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:00.871456 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:00.871419 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:01.534739 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:01.534705 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:02.150862 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:02.150832 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:02.447882 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:02.447794 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:02.554710 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:02.554682 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:02.894611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:02.894584 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:03.054770 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:03.054742 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:03.719207 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:03.719151 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:03.819621 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:03.819577 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:05.454487 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:05.454452 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9"] Sep 16 01:37:05.466406 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:05.466376 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2"] Sep 16 01:37:05.476746 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:05.476716 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8"] Sep 16 01:37:05.533450 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:05.533422 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:09.394894 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:09.394862 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:09.497970 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:09.497940 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:09.595329 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:09.595289 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:09.695261 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:09.695189 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:10.393384 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:10.393354 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:10.516790 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:10.516737 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:10.605673 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:10.605624 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:12.265273 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:12.265245 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:16.284936 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:16.284885 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:16.355090 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:16.355058 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:16.414364 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:16.414333 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:16.474148 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:16.474120 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:16.530905 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:16.530881 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:16.864253 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:16.864214 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:16.962621 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:16.962585 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:17.781989 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:17.781957 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:18.474674 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:18.474643 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:18.576966 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:18.576937 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:18.675291 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:18.675255 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:19.216021 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:19.215984 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:20.116939 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.116905 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:37:20.479820 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.479740 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" podUID="ea1423a7-3e9e-4ed2-ae2d-62530a212c19" containerName="main" containerID="cri-o://120e17f9fd5d9aea87c607f0998d62d60d6995999db88d252234f28255bc668e" gracePeriod=45 Sep 16 01:37:20.490556 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.490524 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" podUID="2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2" containerName="main" containerID="cri-o://f2e1f7f6ab52430ba8b34378c642eb89a2aea988496bfb90f15b5a17aafb6a99" gracePeriod=45 Sep 16 01:37:20.501559 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.501513 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" podUID="8390c3ea-b357-4597-b2a2-8aea50a0b198" containerName="main" containerID="cri-o://6c76960189e69fc1546fec54b76350de5a53b9059984400c5fadae917d10056a" gracePeriod=45 Sep 16 01:37:20.819888 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.819861 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:37:20.823043 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.823020 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:37:20.826401 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.826383 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:37:20.896449 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896415 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-tmp-dir\") pod \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " Sep 16 01:37:20.896641 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896480 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-home\") pod \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " Sep 16 01:37:20.896641 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896530 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-model-cache\") pod \"8390c3ea-b357-4597-b2a2-8aea50a0b198\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " Sep 16 01:37:20.896641 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896562 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-br4bh\" (UniqueName: \"kubernetes.io/projected/8390c3ea-b357-4597-b2a2-8aea50a0b198-kube-api-access-br4bh\") pod \"8390c3ea-b357-4597-b2a2-8aea50a0b198\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " Sep 16 01:37:20.896641 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896591 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-tmp-dir\") pod \"8390c3ea-b357-4597-b2a2-8aea50a0b198\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " Sep 16 01:37:20.896641 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896623 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-tls-certs\") pod \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " Sep 16 01:37:20.896934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896649 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm5tq\" (UniqueName: \"kubernetes.io/projected/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-kube-api-access-pm5tq\") pod \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " Sep 16 01:37:20.896934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896699 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-dshm\") pod \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " Sep 16 01:37:20.896934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896725 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-dshm\") pod \"8390c3ea-b357-4597-b2a2-8aea50a0b198\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " Sep 16 01:37:20.896934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896726 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2" (UID: "2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:20.896934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896749 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-home\") pod \"8390c3ea-b357-4597-b2a2-8aea50a0b198\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " Sep 16 01:37:20.896934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896762 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-home" (OuterVolumeSpecName: "home") pod "ea1423a7-3e9e-4ed2-ae2d-62530a212c19" (UID: "ea1423a7-3e9e-4ed2-ae2d-62530a212c19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:20.896934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896785 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-tls-certs\") pod \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " Sep 16 01:37:20.896934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896789 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-model-cache" (OuterVolumeSpecName: "model-cache") pod "8390c3ea-b357-4597-b2a2-8aea50a0b198" (UID: "8390c3ea-b357-4597-b2a2-8aea50a0b198"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:20.896934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896838 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-dshm\") pod \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " Sep 16 01:37:20.896934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896880 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbslp\" (UniqueName: \"kubernetes.io/projected/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-kube-api-access-tbslp\") pod \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " Sep 16 01:37:20.896934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896910 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-model-cache\") pod \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " Sep 16 01:37:20.896934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896933 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-tmp-dir\") pod \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " Sep 16 01:37:20.897729 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.896964 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-home\") pod \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\" (UID: \"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2\") " Sep 16 01:37:20.897729 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.897029 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-model-cache\") pod \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\" (UID: \"ea1423a7-3e9e-4ed2-ae2d-62530a212c19\") " Sep 16 01:37:20.897729 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.897029 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-home" (OuterVolumeSpecName: "home") pod "8390c3ea-b357-4597-b2a2-8aea50a0b198" (UID: "8390c3ea-b357-4597-b2a2-8aea50a0b198"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:20.897729 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.897061 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8390c3ea-b357-4597-b2a2-8aea50a0b198-tls-certs\") pod \"8390c3ea-b357-4597-b2a2-8aea50a0b198\" (UID: \"8390c3ea-b357-4597-b2a2-8aea50a0b198\") " Sep 16 01:37:20.897729 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.897267 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8390c3ea-b357-4597-b2a2-8aea50a0b198" (UID: "8390c3ea-b357-4597-b2a2-8aea50a0b198"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:20.897729 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.897503 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-tmp-dir\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.897729 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.897520 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-home\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.897729 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.897534 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-model-cache\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.897729 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.897548 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-tmp-dir\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.897729 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.897560 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-home\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.898966 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.898209 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea1423a7-3e9e-4ed2-ae2d-62530a212c19" (UID: "ea1423a7-3e9e-4ed2-ae2d-62530a212c19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:20.898966 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.898428 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2" (UID: "2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:20.898966 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.898615 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-home" (OuterVolumeSpecName: "home") pod "2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2" (UID: "2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:20.898966 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.898789 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea1423a7-3e9e-4ed2-ae2d-62530a212c19" (UID: "ea1423a7-3e9e-4ed2-ae2d-62530a212c19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:20.900471 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.900385 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-dshm" (OuterVolumeSpecName: "dshm") pod "ea1423a7-3e9e-4ed2-ae2d-62530a212c19" (UID: "ea1423a7-3e9e-4ed2-ae2d-62530a212c19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:20.900587 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.900490 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-kube-api-access-tbslp" (OuterVolumeSpecName: "kube-api-access-tbslp") pod "2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2" (UID: "2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2"). InnerVolumeSpecName "kube-api-access-tbslp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:37:20.900782 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.900737 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-dshm" (OuterVolumeSpecName: "dshm") pod "8390c3ea-b357-4597-b2a2-8aea50a0b198" (UID: "8390c3ea-b357-4597-b2a2-8aea50a0b198"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:20.900903 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.900780 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-kube-api-access-pm5tq" (OuterVolumeSpecName: "kube-api-access-pm5tq") pod "ea1423a7-3e9e-4ed2-ae2d-62530a212c19" (UID: "ea1423a7-3e9e-4ed2-ae2d-62530a212c19"). InnerVolumeSpecName "kube-api-access-pm5tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:37:20.901278 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.901249 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea1423a7-3e9e-4ed2-ae2d-62530a212c19" (UID: "ea1423a7-3e9e-4ed2-ae2d-62530a212c19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:37:20.901524 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.901494 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2" (UID: "2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:37:20.901857 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.901838 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8390c3ea-b357-4597-b2a2-8aea50a0b198-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8390c3ea-b357-4597-b2a2-8aea50a0b198" (UID: "8390c3ea-b357-4597-b2a2-8aea50a0b198"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:37:20.901996 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.901981 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-dshm" (OuterVolumeSpecName: "dshm") pod "2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2" (UID: "2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:20.902602 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.902581 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8390c3ea-b357-4597-b2a2-8aea50a0b198-kube-api-access-br4bh" (OuterVolumeSpecName: "kube-api-access-br4bh") pod "8390c3ea-b357-4597-b2a2-8aea50a0b198" (UID: "8390c3ea-b357-4597-b2a2-8aea50a0b198"). InnerVolumeSpecName "kube-api-access-br4bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:37:20.999070 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.999042 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-dshm\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.999070 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.999068 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbslp\" (UniqueName: \"kubernetes.io/projected/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-kube-api-access-tbslp\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.999280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.999078 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-model-cache\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.999280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.999088 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-tmp-dir\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.999280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.999096 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-home\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.999280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.999103 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-model-cache\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.999280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.999110 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8390c3ea-b357-4597-b2a2-8aea50a0b198-tls-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.999280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.999118 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br4bh\" (UniqueName: \"kubernetes.io/projected/8390c3ea-b357-4597-b2a2-8aea50a0b198-kube-api-access-br4bh\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.999280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.999125 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2-tls-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.999280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.999133 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm5tq\" (UniqueName: \"kubernetes.io/projected/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-kube-api-access-pm5tq\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.999280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.999140 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-dshm\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.999280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.999147 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8390c3ea-b357-4597-b2a2-8aea50a0b198-dshm\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:20.999280 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:20.999180 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1423a7-3e9e-4ed2-ae2d-62530a212c19-tls-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:21.523171 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.523124 2558 generic.go:356] "Generic (PLEG): container finished" podID="ea1423a7-3e9e-4ed2-ae2d-62530a212c19" containerID="120e17f9fd5d9aea87c607f0998d62d60d6995999db88d252234f28255bc668e" exitCode=0 Sep 16 01:37:21.523567 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.523185 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" Sep 16 01:37:21.523567 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.523214 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" event={"ID":"ea1423a7-3e9e-4ed2-ae2d-62530a212c19","Type":"ContainerDied","Data":"120e17f9fd5d9aea87c607f0998d62d60d6995999db88d252234f28255bc668e"} Sep 16 01:37:21.523567 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.523270 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9" event={"ID":"ea1423a7-3e9e-4ed2-ae2d-62530a212c19","Type":"ContainerDied","Data":"889d41ceca4e3dd698fdec77192154674a2c706ed2aabcaf9c435b2a9557aa01"} Sep 16 01:37:21.523567 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.523290 2558 scope.go:117] "RemoveContainer" containerID="120e17f9fd5d9aea87c607f0998d62d60d6995999db88d252234f28255bc668e" Sep 16 01:37:21.525034 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.525009 2558 generic.go:356] "Generic (PLEG): container finished" podID="2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2" containerID="f2e1f7f6ab52430ba8b34378c642eb89a2aea988496bfb90f15b5a17aafb6a99" exitCode=0 Sep 16 01:37:21.525180 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.525122 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" event={"ID":"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2","Type":"ContainerDied","Data":"f2e1f7f6ab52430ba8b34378c642eb89a2aea988496bfb90f15b5a17aafb6a99"} Sep 16 01:37:21.525263 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.525179 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" event={"ID":"2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2","Type":"ContainerDied","Data":"a0da232d53f7fec872d4330b77f4c2bfdfa4511602e164b323a7c8ac86ea8853"} Sep 16 01:37:21.525263 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.525233 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2" Sep 16 01:37:21.527315 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.527290 2558 generic.go:356] "Generic (PLEG): container finished" podID="8390c3ea-b357-4597-b2a2-8aea50a0b198" containerID="6c76960189e69fc1546fec54b76350de5a53b9059984400c5fadae917d10056a" exitCode=0 Sep 16 01:37:21.527435 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.527329 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" Sep 16 01:37:21.527435 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.527370 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" event={"ID":"8390c3ea-b357-4597-b2a2-8aea50a0b198","Type":"ContainerDied","Data":"6c76960189e69fc1546fec54b76350de5a53b9059984400c5fadae917d10056a"} Sep 16 01:37:21.527435 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.527420 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8" event={"ID":"8390c3ea-b357-4597-b2a2-8aea50a0b198","Type":"ContainerDied","Data":"e1e698f2b804f2d2390445a256c63cd2aa1319b686a103e6ac10aecbb6c0b8ca"} Sep 16 01:37:21.539454 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.539435 2558 scope.go:117] "RemoveContainer" containerID="120e17f9fd5d9aea87c607f0998d62d60d6995999db88d252234f28255bc668e" Sep 16 01:37:21.539836 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:37:21.539808 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"120e17f9fd5d9aea87c607f0998d62d60d6995999db88d252234f28255bc668e\": container with ID starting with 120e17f9fd5d9aea87c607f0998d62d60d6995999db88d252234f28255bc668e not found: ID does not exist" containerID="120e17f9fd5d9aea87c607f0998d62d60d6995999db88d252234f28255bc668e" Sep 16 01:37:21.539933 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.539848 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"120e17f9fd5d9aea87c607f0998d62d60d6995999db88d252234f28255bc668e"} err="failed to get container status \"120e17f9fd5d9aea87c607f0998d62d60d6995999db88d252234f28255bc668e\": rpc error: code = NotFound desc = could not find container \"120e17f9fd5d9aea87c607f0998d62d60d6995999db88d252234f28255bc668e\": container with ID starting with 120e17f9fd5d9aea87c607f0998d62d60d6995999db88d252234f28255bc668e not found: ID does not exist" Sep 16 01:37:21.539933 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.539870 2558 scope.go:117] "RemoveContainer" containerID="f2e1f7f6ab52430ba8b34378c642eb89a2aea988496bfb90f15b5a17aafb6a99" Sep 16 01:37:21.550012 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.549993 2558 scope.go:117] "RemoveContainer" containerID="f2e1f7f6ab52430ba8b34378c642eb89a2aea988496bfb90f15b5a17aafb6a99" Sep 16 01:37:21.550338 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:37:21.550309 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e1f7f6ab52430ba8b34378c642eb89a2aea988496bfb90f15b5a17aafb6a99\": container with ID starting with f2e1f7f6ab52430ba8b34378c642eb89a2aea988496bfb90f15b5a17aafb6a99 not found: ID does not exist" containerID="f2e1f7f6ab52430ba8b34378c642eb89a2aea988496bfb90f15b5a17aafb6a99" Sep 16 01:37:21.550414 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.550335 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e1f7f6ab52430ba8b34378c642eb89a2aea988496bfb90f15b5a17aafb6a99"} err="failed to get container status \"f2e1f7f6ab52430ba8b34378c642eb89a2aea988496bfb90f15b5a17aafb6a99\": rpc error: code = NotFound desc = could not find container \"f2e1f7f6ab52430ba8b34378c642eb89a2aea988496bfb90f15b5a17aafb6a99\": container with ID starting with f2e1f7f6ab52430ba8b34378c642eb89a2aea988496bfb90f15b5a17aafb6a99 not found: ID does not exist" Sep 16 01:37:21.550414 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.550357 2558 scope.go:117] "RemoveContainer" containerID="6c76960189e69fc1546fec54b76350de5a53b9059984400c5fadae917d10056a" Sep 16 01:37:21.557546 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.557523 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2"] Sep 16 01:37:21.560312 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.560294 2558 scope.go:117] "RemoveContainer" containerID="6c76960189e69fc1546fec54b76350de5a53b9059984400c5fadae917d10056a" Sep 16 01:37:21.560682 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:37:21.560655 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c76960189e69fc1546fec54b76350de5a53b9059984400c5fadae917d10056a\": container with ID starting with 6c76960189e69fc1546fec54b76350de5a53b9059984400c5fadae917d10056a not found: ID does not exist" containerID="6c76960189e69fc1546fec54b76350de5a53b9059984400c5fadae917d10056a" Sep 16 01:37:21.560748 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.560690 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c76960189e69fc1546fec54b76350de5a53b9059984400c5fadae917d10056a"} err="failed to get container status \"6c76960189e69fc1546fec54b76350de5a53b9059984400c5fadae917d10056a\": rpc error: code = NotFound desc = could not find container \"6c76960189e69fc1546fec54b76350de5a53b9059984400c5fadae917d10056a\": container with ID starting with 6c76960189e69fc1546fec54b76350de5a53b9059984400c5fadae917d10056a not found: ID does not exist" Sep 16 01:37:21.561305 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.561284 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-482177/premium-w0-482177-kserve-f77675fd9-dcsl2"] Sep 16 01:37:21.568055 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.568028 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9"] Sep 16 01:37:21.571869 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.571850 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-482177/distinct-w0-482177-kserve-c9bc568d-rpxh9"] Sep 16 01:37:21.582496 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.582474 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8"] Sep 16 01:37:21.586813 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:21.586793 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-482177/simulator-w0-482177-kserve-6d769b855b-sf7h8"] Sep 16 01:37:22.296709 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:22.296665 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2" path="/var/lib/kubelet/pods/2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2/volumes" Sep 16 01:37:22.297119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:22.297103 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8390c3ea-b357-4597-b2a2-8aea50a0b198" path="/var/lib/kubelet/pods/8390c3ea-b357-4597-b2a2-8aea50a0b198/volumes" Sep 16 01:37:22.297499 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:22.297482 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea1423a7-3e9e-4ed2-ae2d-62530a212c19" path="/var/lib/kubelet/pods/ea1423a7-3e9e-4ed2-ae2d-62530a212c19/volumes" Sep 16 01:37:22.669664 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:22.669639 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:22.769805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:22.769777 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:22.936921 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:22.936828 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7"] Sep 16 01:37:22.937109 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:22.937076 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerName="istio-proxy" containerID="cri-o://5572e10bc359aca4f5e46c2f9083b7b301cbaf174855830986d4ba99286d297a" gracePeriod=30 Sep 16 01:37:22.980179 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:22.980126 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:23.890667 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:23.890633 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:23.945236 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:23.945206 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:24.845031 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:24.844997 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:25.266646 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:25.266595 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:25.366845 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:25.366813 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:25.530448 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:25.530363 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:25.869970 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:25.869936 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:26.068097 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:26.068064 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:28.193426 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.193403 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:37:28.376124 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376092 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9h7j\" (UniqueName: \"kubernetes.io/projected/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-kube-api-access-x9h7j\") pod \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " Sep 16 01:37:28.376331 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376133 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-podinfo\") pod \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " Sep 16 01:37:28.376331 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376212 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-workload-socket\") pod \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " Sep 16 01:37:28.376331 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376287 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-envoy\") pod \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " Sep 16 01:37:28.376500 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376339 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-token\") pod \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " Sep 16 01:37:28.376500 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376383 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-data\") pod \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " Sep 16 01:37:28.376500 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376453 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istiod-ca-cert\") pod \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " Sep 16 01:37:28.376500 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376482 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-credential-socket\") pod \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " Sep 16 01:37:28.376709 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376508 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-workload-certs\") pod \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\" (UID: \"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b\") " Sep 16 01:37:28.376709 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376598 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" (UID: "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:28.376709 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376618 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-data" (OuterVolumeSpecName: "istio-data") pod "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" (UID: "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:28.376885 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376815 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" (UID: "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:28.376951 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376932 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-credential-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:28.377005 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376953 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-workload-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:28.377005 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.376968 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-data\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:28.377216 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.377192 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" (UID: "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:28.377446 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.377425 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" (UID: "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:37:28.378689 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.378663 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-token" (OuterVolumeSpecName: "istio-token") pod "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" (UID: "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:37:28.378689 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.378684 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" (UID: "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:37:28.378925 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.378902 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-kube-api-access-x9h7j" (OuterVolumeSpecName: "kube-api-access-x9h7j") pod "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" (UID: "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b"). InnerVolumeSpecName "kube-api-access-x9h7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:37:28.378925 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.378906 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" (UID: "eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:37:28.477864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.477837 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-envoy\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:28.477864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.477863 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-token\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:28.478018 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.477873 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istiod-ca-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:28.478018 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.477881 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-workload-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:28.478018 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.477889 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9h7j\" (UniqueName: \"kubernetes.io/projected/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-kube-api-access-x9h7j\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:28.478018 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.477898 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b-istio-podinfo\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:37:28.575338 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.575308 2558 generic.go:356] "Generic (PLEG): container finished" podID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerID="5572e10bc359aca4f5e46c2f9083b7b301cbaf174855830986d4ba99286d297a" exitCode=0 Sep 16 01:37:28.575551 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.575373 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" Sep 16 01:37:28.575551 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.575399 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" event={"ID":"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b","Type":"ContainerDied","Data":"5572e10bc359aca4f5e46c2f9083b7b301cbaf174855830986d4ba99286d297a"} Sep 16 01:37:28.575551 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.575460 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7" event={"ID":"eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b","Type":"ContainerDied","Data":"bb8a90c737f96aa458830e50b75b01a738cdadc216c69a937f94ba8519b46e8f"} Sep 16 01:37:28.575551 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.575480 2558 scope.go:117] "RemoveContainer" containerID="5572e10bc359aca4f5e46c2f9083b7b301cbaf174855830986d4ba99286d297a" Sep 16 01:37:28.593977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.593956 2558 scope.go:117] "RemoveContainer" containerID="7159626a6b55491dc0ae0d4f669b90f723f491358faec067d571de4473c964b4" Sep 16 01:37:28.603071 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.603048 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7"] Sep 16 01:37:28.603611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.603562 2558 scope.go:117] "RemoveContainer" containerID="5572e10bc359aca4f5e46c2f9083b7b301cbaf174855830986d4ba99286d297a" Sep 16 01:37:28.604094 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:37:28.604070 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5572e10bc359aca4f5e46c2f9083b7b301cbaf174855830986d4ba99286d297a\": container with ID starting with 5572e10bc359aca4f5e46c2f9083b7b301cbaf174855830986d4ba99286d297a not found: ID does not exist" containerID="5572e10bc359aca4f5e46c2f9083b7b301cbaf174855830986d4ba99286d297a" Sep 16 01:37:28.604225 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.604098 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5572e10bc359aca4f5e46c2f9083b7b301cbaf174855830986d4ba99286d297a"} err="failed to get container status \"5572e10bc359aca4f5e46c2f9083b7b301cbaf174855830986d4ba99286d297a\": rpc error: code = NotFound desc = could not find container \"5572e10bc359aca4f5e46c2f9083b7b301cbaf174855830986d4ba99286d297a\": container with ID starting with 5572e10bc359aca4f5e46c2f9083b7b301cbaf174855830986d4ba99286d297a not found: ID does not exist" Sep 16 01:37:28.604225 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.604117 2558 scope.go:117] "RemoveContainer" containerID="7159626a6b55491dc0ae0d4f669b90f723f491358faec067d571de4473c964b4" Sep 16 01:37:28.604451 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:37:28.604434 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7159626a6b55491dc0ae0d4f669b90f723f491358faec067d571de4473c964b4\": container with ID starting with 7159626a6b55491dc0ae0d4f669b90f723f491358faec067d571de4473c964b4 not found: ID does not exist" containerID="7159626a6b55491dc0ae0d4f669b90f723f491358faec067d571de4473c964b4" Sep 16 01:37:28.604495 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.604455 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7159626a6b55491dc0ae0d4f669b90f723f491358faec067d571de4473c964b4"} err="failed to get container status \"7159626a6b55491dc0ae0d4f669b90f723f491358faec067d571de4473c964b4\": rpc error: code = NotFound desc = could not find container \"7159626a6b55491dc0ae0d4f669b90f723f491358faec067d571de4473c964b4\": container with ID starting with 7159626a6b55491dc0ae0d4f669b90f723f491358faec067d571de4473c964b4 not found: ID does not exist" Sep 16 01:37:28.606481 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.606461 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-482177-openshift-default-6b76b97566-zz7c7"] Sep 16 01:37:28.728558 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:28.728483 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:29.406996 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:29.406961 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:29.505494 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:29.505454 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:30.296041 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:30.296009 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" path="/var/lib/kubelet/pods/eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b/volumes" Sep 16 01:37:30.480315 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:30.480276 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:31.292001 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.291972 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:31.394323 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.394289 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:31.733871 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.733842 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:31.836922 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.836894 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:31.868534 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.868503 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86"] Sep 16 01:37:31.868962 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.868947 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2" containerName="main" Sep 16 01:37:31.869024 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.868963 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2" containerName="main" Sep 16 01:37:31.869024 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.868977 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea1423a7-3e9e-4ed2-ae2d-62530a212c19" containerName="main" Sep 16 01:37:31.869024 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.868983 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea1423a7-3e9e-4ed2-ae2d-62530a212c19" containerName="main" Sep 16 01:37:31.869024 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.868990 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerName="istio-proxy" Sep 16 01:37:31.869024 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.868996 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerName="istio-proxy" Sep 16 01:37:31.869024 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.869006 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8390c3ea-b357-4597-b2a2-8aea50a0b198" containerName="main" Sep 16 01:37:31.869024 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.869011 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8390c3ea-b357-4597-b2a2-8aea50a0b198" containerName="main" Sep 16 01:37:31.869352 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.869083 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerName="istio-proxy" Sep 16 01:37:31.869352 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.869092 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerName="istio-proxy" Sep 16 01:37:31.869352 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.869099 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea1423a7-3e9e-4ed2-ae2d-62530a212c19" containerName="main" Sep 16 01:37:31.869352 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.869105 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba6359e-f8bc-44f4-88c0-19e22ed5d7e2" containerName="main" Sep 16 01:37:31.869352 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.869111 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8390c3ea-b357-4597-b2a2-8aea50a0b198" containerName="main" Sep 16 01:37:31.873515 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.873493 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" Sep 16 01:37:31.876561 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.876532 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-61dff065-dockercfg-5cwdb\"" Sep 16 01:37:31.876561 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.876554 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-61dff065\"" Sep 16 01:37:31.884353 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.884330 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86"] Sep 16 01:37:31.905053 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:31.905024 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:32.008667 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:32.008576 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz9bj\" (UniqueName: \"kubernetes.io/projected/7f743f28-bd78-45f7-9ddc-33b95a5349fb-kube-api-access-xz9bj\") pod \"payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86\" (UID: \"7f743f28-bd78-45f7-9ddc-33b95a5349fb\") " pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" Sep 16 01:37:32.008667 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:32.008636 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f743f28-bd78-45f7-9ddc-33b95a5349fb-plugins-config-volume\") pod \"payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86\" (UID: \"7f743f28-bd78-45f7-9ddc-33b95a5349fb\") " pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" Sep 16 01:37:32.109999 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:32.109961 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz9bj\" (UniqueName: \"kubernetes.io/projected/7f743f28-bd78-45f7-9ddc-33b95a5349fb-kube-api-access-xz9bj\") pod \"payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86\" (UID: \"7f743f28-bd78-45f7-9ddc-33b95a5349fb\") " pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" Sep 16 01:37:32.110140 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:32.110014 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f743f28-bd78-45f7-9ddc-33b95a5349fb-plugins-config-volume\") pod \"payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86\" (UID: \"7f743f28-bd78-45f7-9ddc-33b95a5349fb\") " pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" Sep 16 01:37:32.110736 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:32.110715 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f743f28-bd78-45f7-9ddc-33b95a5349fb-plugins-config-volume\") pod \"payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86\" (UID: \"7f743f28-bd78-45f7-9ddc-33b95a5349fb\") " pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" Sep 16 01:37:32.117355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:32.117333 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz9bj\" (UniqueName: \"kubernetes.io/projected/7f743f28-bd78-45f7-9ddc-33b95a5349fb-kube-api-access-xz9bj\") pod \"payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86\" (UID: \"7f743f28-bd78-45f7-9ddc-33b95a5349fb\") " pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" Sep 16 01:37:32.186369 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:32.186337 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" Sep 16 01:37:32.461456 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:32.461391 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86"] Sep 16 01:37:32.464014 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:37:32.463989 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f743f28_bd78_45f7_9ddc_33b95a5349fb.slice/crio-c53cafa3db2c9aa97923372b706cf15d4435fe5ec4a5c3b7ca2bb1f773a1b492 WatchSource:0}: Error finding container c53cafa3db2c9aa97923372b706cf15d4435fe5ec4a5c3b7ca2bb1f773a1b492: Status 404 returned error can't find the container with id c53cafa3db2c9aa97923372b706cf15d4435fe5ec4a5c3b7ca2bb1f773a1b492 Sep 16 01:37:32.606070 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:32.606034 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" event={"ID":"7f743f28-bd78-45f7-9ddc-33b95a5349fb","Type":"ContainerStarted","Data":"a2de584d7a29e771fdf35e10789d3eea4db8e0750aa843669723f78b970e15f4"} Sep 16 01:37:32.606275 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:32.606089 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" event={"ID":"7f743f28-bd78-45f7-9ddc-33b95a5349fb","Type":"ContainerStarted","Data":"c53cafa3db2c9aa97923372b706cf15d4435fe5ec4a5c3b7ca2bb1f773a1b492"} Sep 16 01:37:32.606275 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:32.606256 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" Sep 16 01:37:32.622886 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:32.622836 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" podStartSLOduration=1.622821587 podStartE2EDuration="1.622821587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:37:32.620438934 +0000 UTC m=+1282.872861536" watchObservedRunningTime="2026-09-16 01:37:32.622821587 +0000 UTC m=+1282.875244176" Sep 16 01:37:43.613751 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:43.613720 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" Sep 16 01:37:51.013392 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:51.013354 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:37:53.195622 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:53.195559 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:53.804864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:53.804832 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:54.452914 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:54.452884 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:54.552543 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:54.552516 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:55.243129 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:55.243099 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:55.340484 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:55.340456 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:55.440688 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:55.440653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:37:55.728973 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:37:55.728945 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:01.303787 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:01.303756 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:01.360726 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:01.360694 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:02.102569 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.102537 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw"] Sep 16 01:38:02.102987 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.102975 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerName="istio-proxy" Sep 16 01:38:02.103034 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.102988 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="eabbd6e5-e0cc-4b5d-b1a0-e1180476a99b" containerName="istio-proxy" Sep 16 01:38:02.106277 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.106254 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.109039 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.109015 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c0ec73\"/\"default-dockercfg-p4jqz\"" Sep 16 01:38:02.109176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.109050 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c0ec73\"/\"auth-test-model-c0ec73-kserve-self-signed-certs\"" Sep 16 01:38:02.109176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.109059 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c0ec73\"/\"openshift-service-ca.crt\"" Sep 16 01:38:02.109176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.109097 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c0ec73\"/\"kube-root-ca.crt\"" Sep 16 01:38:02.115934 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.115910 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw"] Sep 16 01:38:02.129031 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.129004 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:02.180267 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.180240 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-tmp-dir\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.180395 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.180279 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88ce1004-8ab1-415f-9be0-c8ea53e54a75-tls-certs\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.180395 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.180301 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfzg7\" (UniqueName: \"kubernetes.io/projected/88ce1004-8ab1-415f-9be0-c8ea53e54a75-kube-api-access-bfzg7\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.180521 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.180415 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-model-cache\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.180521 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.180469 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-dshm\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.180521 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.180496 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-home\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.281258 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.281230 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-tmp-dir\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.281397 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.281275 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88ce1004-8ab1-415f-9be0-c8ea53e54a75-tls-certs\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.281397 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.281308 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfzg7\" (UniqueName: \"kubernetes.io/projected/88ce1004-8ab1-415f-9be0-c8ea53e54a75-kube-api-access-bfzg7\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.281397 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.281362 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-model-cache\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.281568 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.281403 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-dshm\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.281568 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.281424 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-home\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.281712 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.281686 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-tmp-dir\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.281832 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.281814 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-home\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.282001 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.281803 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-model-cache\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.283840 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.283818 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-dshm\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.284569 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.284540 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88ce1004-8ab1-415f-9be0-c8ea53e54a75-tls-certs\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.289313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.289282 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfzg7\" (UniqueName: \"kubernetes.io/projected/88ce1004-8ab1-415f-9be0-c8ea53e54a75-kube-api-access-bfzg7\") pod \"auth-test-model-c0ec73-kserve-68899c5759-t2jzw\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.417277 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.417248 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:02.422878 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.422846 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:02.694699 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.694671 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw"] Sep 16 01:38:02.696273 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:38:02.696249 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88ce1004_8ab1_415f_9be0_c8ea53e54a75.slice/crio-d79cdfaf5d607c01affc6894b9cabd8ac9bfe96a7b4f3a07559730db98022e95 WatchSource:0}: Error finding container d79cdfaf5d607c01affc6894b9cabd8ac9bfe96a7b4f3a07559730db98022e95: Status 404 returned error can't find the container with id d79cdfaf5d607c01affc6894b9cabd8ac9bfe96a7b4f3a07559730db98022e95 Sep 16 01:38:02.798944 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.798910 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" event={"ID":"88ce1004-8ab1-415f-9be0-c8ea53e54a75","Type":"ContainerStarted","Data":"a131a0788fc8c3262630065cf4f0d9309ef57abbbd51c132edc793ed53753498"} Sep 16 01:38:02.798944 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.798954 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" event={"ID":"88ce1004-8ab1-415f-9be0-c8ea53e54a75","Type":"ContainerStarted","Data":"d79cdfaf5d607c01affc6894b9cabd8ac9bfe96a7b4f3a07559730db98022e95"} Sep 16 01:38:02.816178 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:02.814382 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" podStartSLOduration=0.814360366 podStartE2EDuration="814.360366ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:38:02.812834704 +0000 UTC m=+1313.065257284" watchObservedRunningTime="2026-09-16 01:38:02.814360366 +0000 UTC m=+1313.066782956" Sep 16 01:38:03.361615 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:03.361580 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:04.166425 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:04.166383 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:04.790572 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:04.790535 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:05.710333 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:05.710305 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:08.122033 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:08.121993 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:38:11.395639 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:11.395604 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:12.256935 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.256907 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:12.281890 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.281838 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86"] Sep 16 01:38:12.282195 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.282126 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" podUID="7f743f28-bd78-45f7-9ddc-33b95a5349fb" containerName="payload-processing" containerID="cri-o://a2de584d7a29e771fdf35e10789d3eea4db8e0750aa843669723f78b970e15f4" gracePeriod=30 Sep 16 01:38:12.357567 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.357538 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:12.418392 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.418364 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:12.418748 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.418405 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:12.431660 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.431629 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:12.440182 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.440100 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:12.491568 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.491538 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:12.539786 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.539732 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" Sep 16 01:38:12.677421 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.677391 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz9bj\" (UniqueName: \"kubernetes.io/projected/7f743f28-bd78-45f7-9ddc-33b95a5349fb-kube-api-access-xz9bj\") pod \"7f743f28-bd78-45f7-9ddc-33b95a5349fb\" (UID: \"7f743f28-bd78-45f7-9ddc-33b95a5349fb\") " Sep 16 01:38:12.677421 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.677425 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f743f28-bd78-45f7-9ddc-33b95a5349fb-plugins-config-volume\") pod \"7f743f28-bd78-45f7-9ddc-33b95a5349fb\" (UID: \"7f743f28-bd78-45f7-9ddc-33b95a5349fb\") " Sep 16 01:38:12.677839 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.677813 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f743f28-bd78-45f7-9ddc-33b95a5349fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f743f28-bd78-45f7-9ddc-33b95a5349fb" (UID: "7f743f28-bd78-45f7-9ddc-33b95a5349fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:38:12.679560 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.679538 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f743f28-bd78-45f7-9ddc-33b95a5349fb-kube-api-access-xz9bj" (OuterVolumeSpecName: "kube-api-access-xz9bj") pod "7f743f28-bd78-45f7-9ddc-33b95a5349fb" (UID: "7f743f28-bd78-45f7-9ddc-33b95a5349fb"). InnerVolumeSpecName "kube-api-access-xz9bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:38:12.778370 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.778301 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz9bj\" (UniqueName: \"kubernetes.io/projected/7f743f28-bd78-45f7-9ddc-33b95a5349fb-kube-api-access-xz9bj\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:38:12.778370 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.778326 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f743f28-bd78-45f7-9ddc-33b95a5349fb-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:38:12.845458 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.845410 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:12.860383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.860261 2558 generic.go:356] "Generic (PLEG): container finished" podID="7f743f28-bd78-45f7-9ddc-33b95a5349fb" containerID="a2de584d7a29e771fdf35e10789d3eea4db8e0750aa843669723f78b970e15f4" exitCode=0 Sep 16 01:38:12.860383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.860327 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" Sep 16 01:38:12.860383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.860352 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" event={"ID":"7f743f28-bd78-45f7-9ddc-33b95a5349fb","Type":"ContainerDied","Data":"a2de584d7a29e771fdf35e10789d3eea4db8e0750aa843669723f78b970e15f4"} Sep 16 01:38:12.860649 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.860417 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86" event={"ID":"7f743f28-bd78-45f7-9ddc-33b95a5349fb","Type":"ContainerDied","Data":"c53cafa3db2c9aa97923372b706cf15d4435fe5ec4a5c3b7ca2bb1f773a1b492"} Sep 16 01:38:12.860649 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.860483 2558 scope.go:117] "RemoveContainer" containerID="a2de584d7a29e771fdf35e10789d3eea4db8e0750aa843669723f78b970e15f4" Sep 16 01:38:12.872773 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.872740 2558 scope.go:117] "RemoveContainer" containerID="a2de584d7a29e771fdf35e10789d3eea4db8e0750aa843669723f78b970e15f4" Sep 16 01:38:12.873100 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:38:12.873077 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2de584d7a29e771fdf35e10789d3eea4db8e0750aa843669723f78b970e15f4\": container with ID starting with a2de584d7a29e771fdf35e10789d3eea4db8e0750aa843669723f78b970e15f4 not found: ID does not exist" containerID="a2de584d7a29e771fdf35e10789d3eea4db8e0750aa843669723f78b970e15f4" Sep 16 01:38:12.873203 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.873109 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2de584d7a29e771fdf35e10789d3eea4db8e0750aa843669723f78b970e15f4"} err="failed to get container status \"a2de584d7a29e771fdf35e10789d3eea4db8e0750aa843669723f78b970e15f4\": rpc error: code = NotFound desc = could not find container \"a2de584d7a29e771fdf35e10789d3eea4db8e0750aa843669723f78b970e15f4\": container with ID starting with a2de584d7a29e771fdf35e10789d3eea4db8e0750aa843669723f78b970e15f4 not found: ID does not exist" Sep 16 01:38:12.876230 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.876211 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:38:12.883383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.883363 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86"] Sep 16 01:38:12.891764 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.891739 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61dff065-7f4fb559fc-ptb86"] Sep 16 01:38:12.908406 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:12.908385 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:13.568569 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:13.568516 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:14.299480 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:14.297244 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f743f28-bd78-45f7-9ddc-33b95a5349fb" path="/var/lib/kubelet/pods/7f743f28-bd78-45f7-9ddc-33b95a5349fb/volumes" Sep 16 01:38:14.325346 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:14.325315 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:18.467036 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:18.466999 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:18.566955 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:18.566920 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:18.736492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:18.736403 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:19.089062 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:19.088964 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:19.185178 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:19.185125 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:19.295097 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:19.295063 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:24.933482 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:24.933445 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:25.039869 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:25.039837 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:25.132089 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:25.132061 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:25.293805 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:25.293730 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:29.419919 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:29.419885 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:29.953266 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:29.953230 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:30.834474 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:30.834441 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:30.974097 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:30.974062 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:31.150970 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:31.150943 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:31.550773 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:31.550656 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:31.976536 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:31.976491 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:32.276377 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:32.276301 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:32.374152 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:32.374119 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:32.988282 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:32.988225 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:34.424681 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:34.424648 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27"] Sep 16 01:38:34.502281 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:34.502249 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:35.786413 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:35.786385 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:35.882455 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:35.882421 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:36.666452 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:36.666422 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:37.505685 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:37.505652 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:37.603795 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:37.603766 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:37.632827 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:37.632797 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:38:37.929778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:37.929738 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:38.030387 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.030353 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:38.085680 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.085647 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt"] Sep 16 01:38:38.086058 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.086046 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f743f28-bd78-45f7-9ddc-33b95a5349fb" containerName="payload-processing" Sep 16 01:38:38.086106 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.086062 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f743f28-bd78-45f7-9ddc-33b95a5349fb" containerName="payload-processing" Sep 16 01:38:38.086146 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.086138 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f743f28-bd78-45f7-9ddc-33b95a5349fb" containerName="payload-processing" Sep 16 01:38:38.090540 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.090524 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" Sep 16 01:38:38.092717 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.092694 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-4bc0fb\"" Sep 16 01:38:38.092844 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.092752 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-4bc0fb-dockercfg-7rgbg\"" Sep 16 01:38:38.100176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.100137 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt"] Sep 16 01:38:38.105240 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.105218 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:38.203111 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.203028 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cwwh\" (UniqueName: \"kubernetes.io/projected/aa6eeecb-cd72-4ce8-82a3-96026d80131f-kube-api-access-8cwwh\") pod \"payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt\" (UID: \"aa6eeecb-cd72-4ce8-82a3-96026d80131f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" Sep 16 01:38:38.203312 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.203231 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6eeecb-cd72-4ce8-82a3-96026d80131f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt\" (UID: \"aa6eeecb-cd72-4ce8-82a3-96026d80131f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" Sep 16 01:38:38.304362 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.304332 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6eeecb-cd72-4ce8-82a3-96026d80131f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt\" (UID: \"aa6eeecb-cd72-4ce8-82a3-96026d80131f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" Sep 16 01:38:38.304511 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.304377 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cwwh\" (UniqueName: \"kubernetes.io/projected/aa6eeecb-cd72-4ce8-82a3-96026d80131f-kube-api-access-8cwwh\") pod \"payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt\" (UID: \"aa6eeecb-cd72-4ce8-82a3-96026d80131f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" Sep 16 01:38:38.305004 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.304984 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6eeecb-cd72-4ce8-82a3-96026d80131f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt\" (UID: \"aa6eeecb-cd72-4ce8-82a3-96026d80131f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" Sep 16 01:38:38.312175 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.312131 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cwwh\" (UniqueName: \"kubernetes.io/projected/aa6eeecb-cd72-4ce8-82a3-96026d80131f-kube-api-access-8cwwh\") pod \"payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt\" (UID: \"aa6eeecb-cd72-4ce8-82a3-96026d80131f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" Sep 16 01:38:38.402993 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.402966 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" Sep 16 01:38:38.674288 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:38.674256 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt"] Sep 16 01:38:38.675637 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:38:38.675613 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6eeecb_cd72_4ce8_82a3_96026d80131f.slice/crio-1a4c7a8b170a2465e40a4808de97aca164f53e9974a2639208a1241bc25ebc67 WatchSource:0}: Error finding container 1a4c7a8b170a2465e40a4808de97aca164f53e9974a2639208a1241bc25ebc67: Status 404 returned error can't find the container with id 1a4c7a8b170a2465e40a4808de97aca164f53e9974a2639208a1241bc25ebc67 Sep 16 01:38:39.028503 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:39.028404 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" event={"ID":"aa6eeecb-cd72-4ce8-82a3-96026d80131f","Type":"ContainerStarted","Data":"bb22a594fe4e63df4b2aa191553cc1b8f6e7394119cb36bcdfe5b2540fe1b09a"} Sep 16 01:38:39.028503 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:39.028462 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" event={"ID":"aa6eeecb-cd72-4ce8-82a3-96026d80131f","Type":"ContainerStarted","Data":"1a4c7a8b170a2465e40a4808de97aca164f53e9974a2639208a1241bc25ebc67"} Sep 16 01:38:39.028743 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:39.028519 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" Sep 16 01:38:39.047064 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:39.047010 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" podStartSLOduration=1.046991195 podStartE2EDuration="1.046991195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:38:39.044373478 +0000 UTC m=+1349.296796067" watchObservedRunningTime="2026-09-16 01:38:39.046991195 +0000 UTC m=+1349.299413788" Sep 16 01:38:39.852485 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:39.852457 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:45.406904 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.406852 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:45.427386 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.427347 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv"] Sep 16 01:38:45.427636 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.427588 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" podUID="aad009cd-4750-49fb-87db-8e266b539afb" containerName="payload-pre-processing" containerID="cri-o://1f72b5889b8ae795807ca8949e3cf26b2df57e06236a341153e0840af7c011cf" gracePeriod=30 Sep 16 01:38:45.463942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.463909 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:45.562233 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.562196 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:45.618333 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.618117 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:45.698878 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.698852 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" Sep 16 01:38:45.778251 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.778222 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqvvh\" (UniqueName: \"kubernetes.io/projected/aad009cd-4750-49fb-87db-8e266b539afb-kube-api-access-xqvvh\") pod \"aad009cd-4750-49fb-87db-8e266b539afb\" (UID: \"aad009cd-4750-49fb-87db-8e266b539afb\") " Sep 16 01:38:45.778436 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.778268 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad009cd-4750-49fb-87db-8e266b539afb-plugins-config-volume\") pod \"aad009cd-4750-49fb-87db-8e266b539afb\" (UID: \"aad009cd-4750-49fb-87db-8e266b539afb\") " Sep 16 01:38:45.778682 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.778649 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad009cd-4750-49fb-87db-8e266b539afb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aad009cd-4750-49fb-87db-8e266b539afb" (UID: "aad009cd-4750-49fb-87db-8e266b539afb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:38:45.780495 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.780473 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad009cd-4750-49fb-87db-8e266b539afb-kube-api-access-xqvvh" (OuterVolumeSpecName: "kube-api-access-xqvvh") pod "aad009cd-4750-49fb-87db-8e266b539afb" (UID: "aad009cd-4750-49fb-87db-8e266b539afb"). InnerVolumeSpecName "kube-api-access-xqvvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:38:45.880009 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.879983 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqvvh\" (UniqueName: \"kubernetes.io/projected/aad009cd-4750-49fb-87db-8e266b539afb-kube-api-access-xqvvh\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:38:45.880009 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.880006 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad009cd-4750-49fb-87db-8e266b539afb-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:38:45.937319 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:45.937255 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:46.076373 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:46.076339 2558 generic.go:356] "Generic (PLEG): container finished" podID="aad009cd-4750-49fb-87db-8e266b539afb" containerID="1f72b5889b8ae795807ca8949e3cf26b2df57e06236a341153e0840af7c011cf" exitCode=0 Sep 16 01:38:46.076530 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:46.076382 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" Sep 16 01:38:46.076530 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:46.076415 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" event={"ID":"aad009cd-4750-49fb-87db-8e266b539afb","Type":"ContainerDied","Data":"1f72b5889b8ae795807ca8949e3cf26b2df57e06236a341153e0840af7c011cf"} Sep 16 01:38:46.076530 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:46.076467 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv" event={"ID":"aad009cd-4750-49fb-87db-8e266b539afb","Type":"ContainerDied","Data":"224d5232c5df561c6e14110b10cd9741aff3a4802360340f7172fe229edab0ab"} Sep 16 01:38:46.076530 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:46.076490 2558 scope.go:117] "RemoveContainer" containerID="1f72b5889b8ae795807ca8949e3cf26b2df57e06236a341153e0840af7c011cf" Sep 16 01:38:46.086835 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:46.086817 2558 scope.go:117] "RemoveContainer" containerID="1f72b5889b8ae795807ca8949e3cf26b2df57e06236a341153e0840af7c011cf" Sep 16 01:38:46.087085 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:38:46.087066 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f72b5889b8ae795807ca8949e3cf26b2df57e06236a341153e0840af7c011cf\": container with ID starting with 1f72b5889b8ae795807ca8949e3cf26b2df57e06236a341153e0840af7c011cf not found: ID does not exist" containerID="1f72b5889b8ae795807ca8949e3cf26b2df57e06236a341153e0840af7c011cf" Sep 16 01:38:46.087133 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:46.087092 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f72b5889b8ae795807ca8949e3cf26b2df57e06236a341153e0840af7c011cf"} err="failed to get container status \"1f72b5889b8ae795807ca8949e3cf26b2df57e06236a341153e0840af7c011cf\": rpc error: code = NotFound desc = could not find container \"1f72b5889b8ae795807ca8949e3cf26b2df57e06236a341153e0840af7c011cf\": container with ID starting with 1f72b5889b8ae795807ca8949e3cf26b2df57e06236a341153e0840af7c011cf not found: ID does not exist" Sep 16 01:38:46.115826 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:46.115799 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv"] Sep 16 01:38:46.123446 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:46.123426 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d0c9f2-78c7c6885d-hktrv"] Sep 16 01:38:46.207311 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:46.207248 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:46.295062 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:46.295026 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad009cd-4750-49fb-87db-8e266b539afb" path="/var/lib/kubelet/pods/aad009cd-4750-49fb-87db-8e266b539afb/volumes" Sep 16 01:38:47.034617 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:47.034584 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:47.706758 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:47.706723 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:48.469191 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:48.469141 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:49.172394 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.172363 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:49.444826 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.444730 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" podUID="f5cb3eea-9abe-4b45-a8a1-3d0afca700d0" containerName="main" containerID="cri-o://b6f90d764a6cbab2b6beed9f998a33d37c0c29d829de72be853d024ec6830cc7" gracePeriod=45 Sep 16 01:38:49.705820 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.705798 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:38:49.823311 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.823272 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-dshm\") pod \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " Sep 16 01:38:49.823488 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.823322 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-model-cache\") pod \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " Sep 16 01:38:49.823488 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.823385 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-home\") pod \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " Sep 16 01:38:49.823488 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.823471 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zvtr\" (UniqueName: \"kubernetes.io/projected/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-kube-api-access-5zvtr\") pod \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " Sep 16 01:38:49.823638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.823516 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-tls-certs\") pod \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " Sep 16 01:38:49.823638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.823545 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-tmp-dir\") pod \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\" (UID: \"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0\") " Sep 16 01:38:49.823753 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.823689 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5cb3eea-9abe-4b45-a8a1-3d0afca700d0" (UID: "f5cb3eea-9abe-4b45-a8a1-3d0afca700d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:38:49.823753 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.823712 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-home" (OuterVolumeSpecName: "home") pod "f5cb3eea-9abe-4b45-a8a1-3d0afca700d0" (UID: "f5cb3eea-9abe-4b45-a8a1-3d0afca700d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:38:49.823945 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.823914 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5cb3eea-9abe-4b45-a8a1-3d0afca700d0" (UID: "f5cb3eea-9abe-4b45-a8a1-3d0afca700d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:38:49.824054 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.824016 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-model-cache\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.824054 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.824034 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-home\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.824054 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.824048 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-tmp-dir\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.825770 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.825746 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-dshm" (OuterVolumeSpecName: "dshm") pod "f5cb3eea-9abe-4b45-a8a1-3d0afca700d0" (UID: "f5cb3eea-9abe-4b45-a8a1-3d0afca700d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:38:49.825770 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.825761 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5cb3eea-9abe-4b45-a8a1-3d0afca700d0" (UID: "f5cb3eea-9abe-4b45-a8a1-3d0afca700d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:38:49.825921 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.825853 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-kube-api-access-5zvtr" (OuterVolumeSpecName: "kube-api-access-5zvtr") pod "f5cb3eea-9abe-4b45-a8a1-3d0afca700d0" (UID: "f5cb3eea-9abe-4b45-a8a1-3d0afca700d0"). InnerVolumeSpecName "kube-api-access-5zvtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:38:49.924995 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.924963 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-dshm\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.924995 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.924990 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zvtr\" (UniqueName: \"kubernetes.io/projected/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-kube-api-access-5zvtr\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.924995 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:49.925000 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0-tls-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:38:50.037110 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:50.037033 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" Sep 16 01:38:50.102923 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:50.102885 2558 generic.go:356] "Generic (PLEG): container finished" podID="f5cb3eea-9abe-4b45-a8a1-3d0afca700d0" containerID="b6f90d764a6cbab2b6beed9f998a33d37c0c29d829de72be853d024ec6830cc7" exitCode=0 Sep 16 01:38:50.103100 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:50.102932 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" Sep 16 01:38:50.103100 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:50.102968 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" event={"ID":"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0","Type":"ContainerDied","Data":"b6f90d764a6cbab2b6beed9f998a33d37c0c29d829de72be853d024ec6830cc7"} Sep 16 01:38:50.103100 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:50.103025 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27" event={"ID":"f5cb3eea-9abe-4b45-a8a1-3d0afca700d0","Type":"ContainerDied","Data":"b00c8c3c624fac676cff9fd8781e7f64a16b825b4882b5d71eaf883d89f973e9"} Sep 16 01:38:50.103100 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:50.103050 2558 scope.go:117] "RemoveContainer" containerID="b6f90d764a6cbab2b6beed9f998a33d37c0c29d829de72be853d024ec6830cc7" Sep 16 01:38:50.114687 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:50.114669 2558 scope.go:117] "RemoveContainer" containerID="b6f90d764a6cbab2b6beed9f998a33d37c0c29d829de72be853d024ec6830cc7" Sep 16 01:38:50.114976 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:38:50.114954 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f90d764a6cbab2b6beed9f998a33d37c0c29d829de72be853d024ec6830cc7\": container with ID starting with b6f90d764a6cbab2b6beed9f998a33d37c0c29d829de72be853d024ec6830cc7 not found: ID does not exist" containerID="b6f90d764a6cbab2b6beed9f998a33d37c0c29d829de72be853d024ec6830cc7" Sep 16 01:38:50.115033 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:50.114988 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f90d764a6cbab2b6beed9f998a33d37c0c29d829de72be853d024ec6830cc7"} err="failed to get container status \"b6f90d764a6cbab2b6beed9f998a33d37c0c29d829de72be853d024ec6830cc7\": rpc error: code = NotFound desc = could not find container \"b6f90d764a6cbab2b6beed9f998a33d37c0c29d829de72be853d024ec6830cc7\": container with ID starting with b6f90d764a6cbab2b6beed9f998a33d37c0c29d829de72be853d024ec6830cc7 not found: ID does not exist" Sep 16 01:38:50.123614 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:50.123591 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27"] Sep 16 01:38:50.126900 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:50.126873 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/premium-w3-d0c9f2-kserve-6774c46b69-94w27"] Sep 16 01:38:50.297566 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:50.297489 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5cb3eea-9abe-4b45-a8a1-3d0afca700d0" path="/var/lib/kubelet/pods/f5cb3eea-9abe-4b45-a8a1-3d0afca700d0/volumes" Sep 16 01:38:50.539886 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:50.539857 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:50.901279 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:50.901246 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:51.298127 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:51.298048 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:51.628689 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:51.628663 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:52.582437 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:52.582405 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:52.682485 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:52.682452 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:52.866856 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:52.866827 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:53.363848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:53.363818 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:53.998939 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:53.998911 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:54.092207 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:54.092153 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:54.760687 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:54.760655 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:55.592708 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:55.592678 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:55.694518 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:55.694464 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:55.939966 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:55.939937 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:56.144364 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:56.144333 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:56.241567 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:56.241493 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:38:59.229984 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:38:59.229950 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:02.716326 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:02.716289 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:39:10.573402 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:10.573367 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:10.672703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:10.672669 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:12.003672 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:12.003636 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:16.500872 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:16.500839 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:17.914988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:17.914958 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:18.019553 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:18.019519 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:18.564952 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:18.564908 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:19.100546 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.100514 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:19.269575 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.269536 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:19.886777 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.886742 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:19.920068 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.920041 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:39:19.933888 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.933861 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9135-b8e79a"] Sep 16 01:39:19.934295 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.934282 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad009cd-4750-49fb-87db-8e266b539afb" containerName="payload-pre-processing" Sep 16 01:39:19.934346 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.934300 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad009cd-4750-49fb-87db-8e266b539afb" containerName="payload-pre-processing" Sep 16 01:39:19.934346 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.934314 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5cb3eea-9abe-4b45-a8a1-3d0afca700d0" containerName="main" Sep 16 01:39:19.934346 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.934320 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5cb3eea-9abe-4b45-a8a1-3d0afca700d0" containerName="main" Sep 16 01:39:19.934446 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.934420 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad009cd-4750-49fb-87db-8e266b539afb" containerName="payload-pre-processing" Sep 16 01:39:19.934446 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.934430 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5cb3eea-9abe-4b45-a8a1-3d0afca700d0" containerName="main" Sep 16 01:39:19.938767 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.938751 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-b8e79a" Sep 16 01:39:19.941179 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.941132 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6n2fb\"" Sep 16 01:39:19.942933 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:19.942907 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9135-b8e79a"] Sep 16 01:39:20.001298 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:20.001271 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkqmz\" (UniqueName: \"kubernetes.io/projected/c0aa8059-b2c4-46d3-8226-af739ab13efb-kube-api-access-kkqmz\") pod \"test-curl-9135-b8e79a\" (UID: \"c0aa8059-b2c4-46d3-8226-af739ab13efb\") " pod="opendatahub/test-curl-9135-b8e79a" Sep 16 01:39:20.102872 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:20.102835 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkqmz\" (UniqueName: \"kubernetes.io/projected/c0aa8059-b2c4-46d3-8226-af739ab13efb-kube-api-access-kkqmz\") pod \"test-curl-9135-b8e79a\" (UID: \"c0aa8059-b2c4-46d3-8226-af739ab13efb\") " pod="opendatahub/test-curl-9135-b8e79a" Sep 16 01:39:20.110458 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:20.110434 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkqmz\" (UniqueName: \"kubernetes.io/projected/c0aa8059-b2c4-46d3-8226-af739ab13efb-kube-api-access-kkqmz\") pod \"test-curl-9135-b8e79a\" (UID: \"c0aa8059-b2c4-46d3-8226-af739ab13efb\") " pod="opendatahub/test-curl-9135-b8e79a" Sep 16 01:39:20.249858 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:20.249788 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-b8e79a" Sep 16 01:39:20.527238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:20.527211 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9135-b8e79a"] Sep 16 01:39:20.529733 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:39:20.529705 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0aa8059_b2c4_46d3_8226_af739ab13efb.slice/crio-75137a8a3daaea9db8635f2b253ba24de0d48befa4ddd750e12974cf2b86e55e WatchSource:0}: Error finding container 75137a8a3daaea9db8635f2b253ba24de0d48befa4ddd750e12974cf2b86e55e: Status 404 returned error can't find the container with id 75137a8a3daaea9db8635f2b253ba24de0d48befa4ddd750e12974cf2b86e55e Sep 16 01:39:20.531928 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:20.531911 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 01:39:21.296791 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:21.296753 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9135-b8e79a" event={"ID":"c0aa8059-b2c4-46d3-8226-af739ab13efb","Type":"ContainerStarted","Data":"3de7d8be5e952b235978526c8283db886123e66006549b8c238e1c35476076ee"} Sep 16 01:39:21.297142 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:21.296804 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9135-b8e79a" event={"ID":"c0aa8059-b2c4-46d3-8226-af739ab13efb","Type":"ContainerStarted","Data":"75137a8a3daaea9db8635f2b253ba24de0d48befa4ddd750e12974cf2b86e55e"} Sep 16 01:39:21.310778 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:21.310738 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9135-b8e79a" podStartSLOduration=1.7003012370000001 podStartE2EDuration="2.310722876s" totalImagesPullingTime="610.421639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:39:21.308642112 +0000 UTC m=+1391.561064700" watchObservedRunningTime="2026-09-16 01:39:21.310722876 +0000 UTC m=+1391.563145464" Sep 16 01:39:21.915985 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:21.915951 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9135-b8e79a"] Sep 16 01:39:21.919858 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:21.919833 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9135-b8e79a"] Sep 16 01:39:22.302111 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:39:22.302042 2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9135-b8e79a" container="test-curl-9135-b8e79a" Sep 16 01:39:23.307378 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:23.307337 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9135-b8e79a" podUID="c0aa8059-b2c4-46d3-8226-af739ab13efb" containerName="test-curl-9135-b8e79a" containerID="cri-o://3de7d8be5e952b235978526c8283db886123e66006549b8c238e1c35476076ee" gracePeriod=2 Sep 16 01:39:23.507562 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:23.507532 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:23.608288 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:23.608214 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:25.476708 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.476685 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-b8e79a" Sep 16 01:39:25.552435 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.552409 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkqmz\" (UniqueName: \"kubernetes.io/projected/c0aa8059-b2c4-46d3-8226-af739ab13efb-kube-api-access-kkqmz\") pod \"c0aa8059-b2c4-46d3-8226-af739ab13efb\" (UID: \"c0aa8059-b2c4-46d3-8226-af739ab13efb\") " Sep 16 01:39:25.554680 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.554657 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0aa8059-b2c4-46d3-8226-af739ab13efb-kube-api-access-kkqmz" (OuterVolumeSpecName: "kube-api-access-kkqmz") pod "c0aa8059-b2c4-46d3-8226-af739ab13efb" (UID: "c0aa8059-b2c4-46d3-8226-af739ab13efb"). InnerVolumeSpecName "kube-api-access-kkqmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:39:25.608538 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.608464 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9135-c7c45a"] Sep 16 01:39:25.608970 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.608951 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0aa8059-b2c4-46d3-8226-af739ab13efb" containerName="test-curl-9135-b8e79a" Sep 16 01:39:25.609055 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.608972 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0aa8059-b2c4-46d3-8226-af739ab13efb" containerName="test-curl-9135-b8e79a" Sep 16 01:39:25.609110 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.609065 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0aa8059-b2c4-46d3-8226-af739ab13efb" containerName="test-curl-9135-b8e79a" Sep 16 01:39:25.612121 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.612101 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-c7c45a" Sep 16 01:39:25.617991 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.617961 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9135-c7c45a"] Sep 16 01:39:25.653187 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.653148 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkqmz\" (UniqueName: \"kubernetes.io/projected/c0aa8059-b2c4-46d3-8226-af739ab13efb-kube-api-access-kkqmz\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:39:25.753969 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.753943 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rchm5\" (UniqueName: \"kubernetes.io/projected/81978d16-63f8-4e3b-87f8-e56fd144a374-kube-api-access-rchm5\") pod \"test-curl-9135-c7c45a\" (UID: \"81978d16-63f8-4e3b-87f8-e56fd144a374\") " pod="opendatahub/test-curl-9135-c7c45a" Sep 16 01:39:25.854501 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.854464 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rchm5\" (UniqueName: \"kubernetes.io/projected/81978d16-63f8-4e3b-87f8-e56fd144a374-kube-api-access-rchm5\") pod \"test-curl-9135-c7c45a\" (UID: \"81978d16-63f8-4e3b-87f8-e56fd144a374\") " pod="opendatahub/test-curl-9135-c7c45a" Sep 16 01:39:25.861972 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.861908 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rchm5\" (UniqueName: \"kubernetes.io/projected/81978d16-63f8-4e3b-87f8-e56fd144a374-kube-api-access-rchm5\") pod \"test-curl-9135-c7c45a\" (UID: \"81978d16-63f8-4e3b-87f8-e56fd144a374\") " pod="opendatahub/test-curl-9135-c7c45a" Sep 16 01:39:25.923029 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:25.923007 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-c7c45a" Sep 16 01:39:26.191630 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:26.191607 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9135-c7c45a"] Sep 16 01:39:26.193046 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:39:26.193024 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81978d16_63f8_4e3b_87f8_e56fd144a374.slice/crio-77ac3e29a563a0d27c416e34b7234714951f048e56b89eafcfd2258eb4874a84 WatchSource:0}: Error finding container 77ac3e29a563a0d27c416e34b7234714951f048e56b89eafcfd2258eb4874a84: Status 404 returned error can't find the container with id 77ac3e29a563a0d27c416e34b7234714951f048e56b89eafcfd2258eb4874a84 Sep 16 01:39:26.296393 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:26.296361 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0aa8059-b2c4-46d3-8226-af739ab13efb" path="/var/lib/kubelet/pods/c0aa8059-b2c4-46d3-8226-af739ab13efb/volumes" Sep 16 01:39:26.327625 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:26.327596 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9135-c7c45a" event={"ID":"81978d16-63f8-4e3b-87f8-e56fd144a374","Type":"ContainerStarted","Data":"e61b36953a27b73a996c4a7760a32c3ac2bea220b0b3a973f6e5b6710fa54bd8"} Sep 16 01:39:26.327770 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:26.327637 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9135-c7c45a" event={"ID":"81978d16-63f8-4e3b-87f8-e56fd144a374","Type":"ContainerStarted","Data":"77ac3e29a563a0d27c416e34b7234714951f048e56b89eafcfd2258eb4874a84"} Sep 16 01:39:26.328827 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:26.328803 2558 generic.go:356] "Generic (PLEG): container finished" podID="c0aa8059-b2c4-46d3-8226-af739ab13efb" containerID="3de7d8be5e952b235978526c8283db886123e66006549b8c238e1c35476076ee" exitCode=137 Sep 16 01:39:26.328918 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:26.328843 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-b8e79a" Sep 16 01:39:26.328918 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:26.328896 2558 scope.go:117] "RemoveContainer" containerID="3de7d8be5e952b235978526c8283db886123e66006549b8c238e1c35476076ee" Sep 16 01:39:26.345406 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:26.345372 2558 scope.go:117] "RemoveContainer" containerID="3de7d8be5e952b235978526c8283db886123e66006549b8c238e1c35476076ee" Sep 16 01:39:26.345950 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:39:26.345849 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de7d8be5e952b235978526c8283db886123e66006549b8c238e1c35476076ee\": container with ID starting with 3de7d8be5e952b235978526c8283db886123e66006549b8c238e1c35476076ee not found: ID does not exist" containerID="3de7d8be5e952b235978526c8283db886123e66006549b8c238e1c35476076ee" Sep 16 01:39:26.345950 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:26.345891 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de7d8be5e952b235978526c8283db886123e66006549b8c238e1c35476076ee"} err="failed to get container status \"3de7d8be5e952b235978526c8283db886123e66006549b8c238e1c35476076ee\": rpc error: code = NotFound desc = could not find container \"3de7d8be5e952b235978526c8283db886123e66006549b8c238e1c35476076ee\": container with ID starting with 3de7d8be5e952b235978526c8283db886123e66006549b8c238e1c35476076ee not found: ID does not exist" Sep 16 01:39:26.347489 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:26.347452 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9135-c7c45a" podStartSLOduration=1.347439825 podStartE2EDuration="1.347439825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:39:26.345808244 +0000 UTC m=+1396.598230833" watchObservedRunningTime="2026-09-16 01:39:26.347439825 +0000 UTC m=+1396.599862413" Sep 16 01:39:26.918174 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:26.918099 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9135-c7c45a"] Sep 16 01:39:26.921626 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:26.921599 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9135-c7c45a"] Sep 16 01:39:27.044118 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:27.044088 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9135-278fc7"] Sep 16 01:39:27.044588 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:27.044570 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81978d16-63f8-4e3b-87f8-e56fd144a374" containerName="test-curl-9135-c7c45a" Sep 16 01:39:27.044642 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:27.044591 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="81978d16-63f8-4e3b-87f8-e56fd144a374" containerName="test-curl-9135-c7c45a" Sep 16 01:39:27.044682 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:27.044673 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="81978d16-63f8-4e3b-87f8-e56fd144a374" containerName="test-curl-9135-c7c45a" Sep 16 01:39:27.049342 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:27.049315 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-278fc7" Sep 16 01:39:27.052814 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:27.052790 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9135-278fc7"] Sep 16 01:39:27.168171 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:27.168134 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4t8c\" (UniqueName: \"kubernetes.io/projected/64e95e54-e713-4587-af51-1698b9a6587e-kube-api-access-v4t8c\") pod \"test-curl-9135-278fc7\" (UID: \"64e95e54-e713-4587-af51-1698b9a6587e\") " pod="opendatahub/test-curl-9135-278fc7" Sep 16 01:39:27.269677 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:27.269598 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4t8c\" (UniqueName: \"kubernetes.io/projected/64e95e54-e713-4587-af51-1698b9a6587e-kube-api-access-v4t8c\") pod \"test-curl-9135-278fc7\" (UID: \"64e95e54-e713-4587-af51-1698b9a6587e\") " pod="opendatahub/test-curl-9135-278fc7" Sep 16 01:39:27.277350 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:27.277323 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4t8c\" (UniqueName: \"kubernetes.io/projected/64e95e54-e713-4587-af51-1698b9a6587e-kube-api-access-v4t8c\") pod \"test-curl-9135-278fc7\" (UID: \"64e95e54-e713-4587-af51-1698b9a6587e\") " pod="opendatahub/test-curl-9135-278fc7" Sep 16 01:39:27.336220 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:39:27.336198 2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9135-c7c45a" container="test-curl-9135-c7c45a" Sep 16 01:39:27.360235 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:27.360212 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-278fc7" Sep 16 01:39:27.629436 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:27.629408 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9135-278fc7"] Sep 16 01:39:27.630564 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:39:27.630541 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64e95e54_e713_4587_af51_1698b9a6587e.slice/crio-38bf11099406a2a83c3feaef3bb97f8fbf4418f65ca504a68776335aa86634fd WatchSource:0}: Error finding container 38bf11099406a2a83c3feaef3bb97f8fbf4418f65ca504a68776335aa86634fd: Status 404 returned error can't find the container with id 38bf11099406a2a83c3feaef3bb97f8fbf4418f65ca504a68776335aa86634fd Sep 16 01:39:28.343411 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:28.343375 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9135-278fc7" event={"ID":"64e95e54-e713-4587-af51-1698b9a6587e","Type":"ContainerStarted","Data":"10932351b8d82868f1bec3fe1065fe30e0a0fd4b88ea0e300d1ad6bea1524797"} Sep 16 01:39:28.343803 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:28.343417 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9135-278fc7" event={"ID":"64e95e54-e713-4587-af51-1698b9a6587e","Type":"ContainerStarted","Data":"38bf11099406a2a83c3feaef3bb97f8fbf4418f65ca504a68776335aa86634fd"} Sep 16 01:39:28.343803 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:28.343506 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9135-c7c45a" podUID="81978d16-63f8-4e3b-87f8-e56fd144a374" containerName="test-curl-9135-c7c45a" containerID="cri-o://e61b36953a27b73a996c4a7760a32c3ac2bea220b0b3a973f6e5b6710fa54bd8" gracePeriod=2 Sep 16 01:39:28.355847 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:28.355797 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9135-278fc7" podStartSLOduration=1.355779265 podStartE2EDuration="1.355779265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:39:28.355357922 +0000 UTC m=+1398.607780511" watchObservedRunningTime="2026-09-16 01:39:28.355779265 +0000 UTC m=+1398.608201851" Sep 16 01:39:28.949425 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:28.949389 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9135-278fc7"] Sep 16 01:39:28.952639 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:28.952613 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9135-278fc7"] Sep 16 01:39:29.349778 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:39:29.349747 2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9135-278fc7" container="test-curl-9135-278fc7" Sep 16 01:39:29.572433 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:29.572403 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:29.863115 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:29.863089 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:29.993223 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:29.993188 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:30.225413 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:30.225384 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:30.355731 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:30.355694 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9135-278fc7" podUID="64e95e54-e713-4587-af51-1698b9a6587e" containerName="test-curl-9135-278fc7" containerID="cri-o://10932351b8d82868f1bec3fe1065fe30e0a0fd4b88ea0e300d1ad6bea1524797" gracePeriod=2 Sep 16 01:39:30.511799 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:30.511774 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-c7c45a" Sep 16 01:39:30.602145 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:30.602116 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rchm5\" (UniqueName: \"kubernetes.io/projected/81978d16-63f8-4e3b-87f8-e56fd144a374-kube-api-access-rchm5\") pod \"81978d16-63f8-4e3b-87f8-e56fd144a374\" (UID: \"81978d16-63f8-4e3b-87f8-e56fd144a374\") " Sep 16 01:39:30.604222 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:30.604186 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81978d16-63f8-4e3b-87f8-e56fd144a374-kube-api-access-rchm5" (OuterVolumeSpecName: "kube-api-access-rchm5") pod "81978d16-63f8-4e3b-87f8-e56fd144a374" (UID: "81978d16-63f8-4e3b-87f8-e56fd144a374"). InnerVolumeSpecName "kube-api-access-rchm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:39:30.703758 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:30.703732 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rchm5\" (UniqueName: \"kubernetes.io/projected/81978d16-63f8-4e3b-87f8-e56fd144a374-kube-api-access-rchm5\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:39:31.362732 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:31.362696 2558 generic.go:356] "Generic (PLEG): container finished" podID="81978d16-63f8-4e3b-87f8-e56fd144a374" containerID="e61b36953a27b73a996c4a7760a32c3ac2bea220b0b3a973f6e5b6710fa54bd8" exitCode=137 Sep 16 01:39:31.363148 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:31.362740 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-c7c45a" Sep 16 01:39:31.363148 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:31.362795 2558 scope.go:117] "RemoveContainer" containerID="e61b36953a27b73a996c4a7760a32c3ac2bea220b0b3a973f6e5b6710fa54bd8" Sep 16 01:39:31.373362 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:31.373340 2558 scope.go:117] "RemoveContainer" containerID="e61b36953a27b73a996c4a7760a32c3ac2bea220b0b3a973f6e5b6710fa54bd8" Sep 16 01:39:31.373437 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:39:31.373344 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9135-c7c45a\" is forbidden: User \"system:node:ip-10-0-137-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-55.ec2.internal' and this object" podUID="81978d16-63f8-4e3b-87f8-e56fd144a374" pod="opendatahub/test-curl-9135-c7c45a" Sep 16 01:39:31.373641 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:39:31.373620 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61b36953a27b73a996c4a7760a32c3ac2bea220b0b3a973f6e5b6710fa54bd8\": container with ID starting with e61b36953a27b73a996c4a7760a32c3ac2bea220b0b3a973f6e5b6710fa54bd8 not found: ID does not exist" containerID="e61b36953a27b73a996c4a7760a32c3ac2bea220b0b3a973f6e5b6710fa54bd8" Sep 16 01:39:31.373714 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:31.373665 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61b36953a27b73a996c4a7760a32c3ac2bea220b0b3a973f6e5b6710fa54bd8"} err="failed to get container status \"e61b36953a27b73a996c4a7760a32c3ac2bea220b0b3a973f6e5b6710fa54bd8\": rpc error: code = NotFound desc = could not find container \"e61b36953a27b73a996c4a7760a32c3ac2bea220b0b3a973f6e5b6710fa54bd8\": container with ID starting with e61b36953a27b73a996c4a7760a32c3ac2bea220b0b3a973f6e5b6710fa54bd8 not found: ID does not exist" Sep 16 01:39:32.295424 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:32.295391 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81978d16-63f8-4e3b-87f8-e56fd144a374" path="/var/lib/kubelet/pods/81978d16-63f8-4e3b-87f8-e56fd144a374/volumes" Sep 16 01:39:32.470108 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:32.470076 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:32.547966 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:32.547905 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-278fc7" Sep 16 01:39:32.620447 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:32.620414 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4t8c\" (UniqueName: \"kubernetes.io/projected/64e95e54-e713-4587-af51-1698b9a6587e-kube-api-access-v4t8c\") pod \"64e95e54-e713-4587-af51-1698b9a6587e\" (UID: \"64e95e54-e713-4587-af51-1698b9a6587e\") " Sep 16 01:39:32.622529 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:32.622504 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e95e54-e713-4587-af51-1698b9a6587e-kube-api-access-v4t8c" (OuterVolumeSpecName: "kube-api-access-v4t8c") pod "64e95e54-e713-4587-af51-1698b9a6587e" (UID: "64e95e54-e713-4587-af51-1698b9a6587e"). InnerVolumeSpecName "kube-api-access-v4t8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:39:32.721926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:32.721901 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4t8c\" (UniqueName: \"kubernetes.io/projected/64e95e54-e713-4587-af51-1698b9a6587e-kube-api-access-v4t8c\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:39:33.161521 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:33.161480 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:33.269634 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:33.269592 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:33.376475 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:33.376442 2558 generic.go:356] "Generic (PLEG): container finished" podID="64e95e54-e713-4587-af51-1698b9a6587e" containerID="10932351b8d82868f1bec3fe1065fe30e0a0fd4b88ea0e300d1ad6bea1524797" exitCode=137 Sep 16 01:39:33.376648 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:33.376498 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-278fc7" Sep 16 01:39:33.376648 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:33.376509 2558 scope.go:117] "RemoveContainer" containerID="10932351b8d82868f1bec3fe1065fe30e0a0fd4b88ea0e300d1ad6bea1524797" Sep 16 01:39:33.386799 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:39:33.386672 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9135-278fc7\" is forbidden: User \"system:node:ip-10-0-137-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-55.ec2.internal' and this object" podUID="64e95e54-e713-4587-af51-1698b9a6587e" pod="opendatahub/test-curl-9135-278fc7" Sep 16 01:39:33.386873 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:33.386807 2558 scope.go:117] "RemoveContainer" containerID="10932351b8d82868f1bec3fe1065fe30e0a0fd4b88ea0e300d1ad6bea1524797" Sep 16 01:39:33.387109 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:39:33.387085 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10932351b8d82868f1bec3fe1065fe30e0a0fd4b88ea0e300d1ad6bea1524797\": container with ID starting with 10932351b8d82868f1bec3fe1065fe30e0a0fd4b88ea0e300d1ad6bea1524797 not found: ID does not exist" containerID="10932351b8d82868f1bec3fe1065fe30e0a0fd4b88ea0e300d1ad6bea1524797" Sep 16 01:39:33.387183 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:33.387117 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10932351b8d82868f1bec3fe1065fe30e0a0fd4b88ea0e300d1ad6bea1524797"} err="failed to get container status \"10932351b8d82868f1bec3fe1065fe30e0a0fd4b88ea0e300d1ad6bea1524797\": rpc error: code = NotFound desc = could not find container \"10932351b8d82868f1bec3fe1065fe30e0a0fd4b88ea0e300d1ad6bea1524797\": container with ID starting with 10932351b8d82868f1bec3fe1065fe30e0a0fd4b88ea0e300d1ad6bea1524797 not found: ID does not exist" Sep 16 01:39:34.102876 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:34.102843 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:34.295907 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:34.295878 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64e95e54-e713-4587-af51-1698b9a6587e" path="/var/lib/kubelet/pods/64e95e54-e713-4587-af51-1698b9a6587e/volumes" Sep 16 01:39:34.811672 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:34.811642 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:35.572203 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:35.572149 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:35.629799 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:35.629771 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:35.722418 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:35.722387 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:39:36.072594 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.072565 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:36.112063 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.112032 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr"] Sep 16 01:39:36.112516 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.112500 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64e95e54-e713-4587-af51-1698b9a6587e" containerName="test-curl-9135-278fc7" Sep 16 01:39:36.112567 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.112518 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="64e95e54-e713-4587-af51-1698b9a6587e" containerName="test-curl-9135-278fc7" Sep 16 01:39:36.112607 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.112583 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="64e95e54-e713-4587-af51-1698b9a6587e" containerName="test-curl-9135-278fc7" Sep 16 01:39:36.117330 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.117314 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" Sep 16 01:39:36.119483 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.119460 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3b6e6489-dockercfg-m8hn8\"" Sep 16 01:39:36.119633 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.119487 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3b6e6489\"" Sep 16 01:39:36.123920 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.123892 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr"] Sep 16 01:39:36.138370 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.138346 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx"] Sep 16 01:39:36.142058 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.142032 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" Sep 16 01:39:36.143744 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.143723 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:36.152831 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.152803 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx"] Sep 16 01:39:36.256534 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.256506 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rrq2\" (UniqueName: \"kubernetes.io/projected/bbb68eb0-f56d-4538-9f8a-16eced43811b-kube-api-access-6rrq2\") pod \"payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr\" (UID: \"bbb68eb0-f56d-4538-9f8a-16eced43811b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" Sep 16 01:39:36.256700 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.256543 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv8vl\" (UniqueName: \"kubernetes.io/projected/3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60-kube-api-access-rv8vl\") pod \"payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx\" (UID: \"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60\") " pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" Sep 16 01:39:36.256700 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.256614 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60-plugins-config-volume\") pod \"payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx\" (UID: \"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60\") " pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" Sep 16 01:39:36.256792 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.256697 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb68eb0-f56d-4538-9f8a-16eced43811b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr\" (UID: \"bbb68eb0-f56d-4538-9f8a-16eced43811b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" Sep 16 01:39:36.358283 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.358190 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60-plugins-config-volume\") pod \"payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx\" (UID: \"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60\") " pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" Sep 16 01:39:36.358283 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.358255 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb68eb0-f56d-4538-9f8a-16eced43811b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr\" (UID: \"bbb68eb0-f56d-4538-9f8a-16eced43811b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" Sep 16 01:39:36.358540 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.358290 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rrq2\" (UniqueName: \"kubernetes.io/projected/bbb68eb0-f56d-4538-9f8a-16eced43811b-kube-api-access-6rrq2\") pod \"payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr\" (UID: \"bbb68eb0-f56d-4538-9f8a-16eced43811b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" Sep 16 01:39:36.358540 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.358319 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv8vl\" (UniqueName: \"kubernetes.io/projected/3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60-kube-api-access-rv8vl\") pod \"payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx\" (UID: \"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60\") " pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" Sep 16 01:39:36.359132 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.359105 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb68eb0-f56d-4538-9f8a-16eced43811b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr\" (UID: \"bbb68eb0-f56d-4538-9f8a-16eced43811b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" Sep 16 01:39:36.359281 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.359105 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60-plugins-config-volume\") pod \"payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx\" (UID: \"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60\") " pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" Sep 16 01:39:36.361222 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.361140 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:36.366136 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.366107 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv8vl\" (UniqueName: \"kubernetes.io/projected/3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60-kube-api-access-rv8vl\") pod \"payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx\" (UID: \"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60\") " pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" Sep 16 01:39:36.366658 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.366634 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rrq2\" (UniqueName: \"kubernetes.io/projected/bbb68eb0-f56d-4538-9f8a-16eced43811b-kube-api-access-6rrq2\") pod \"payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr\" (UID: \"bbb68eb0-f56d-4538-9f8a-16eced43811b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" Sep 16 01:39:36.430953 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.430911 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" Sep 16 01:39:36.455744 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.455705 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" Sep 16 01:39:36.725204 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.725178 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr"] Sep 16 01:39:36.726526 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:39:36.726502 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb68eb0_f56d_4538_9f8a_16eced43811b.slice/crio-4e10960e23c7968f3aca3eb52c76cd6db4f2a2d50efe81f579c53ecf629969b5 WatchSource:0}: Error finding container 4e10960e23c7968f3aca3eb52c76cd6db4f2a2d50efe81f579c53ecf629969b5: Status 404 returned error can't find the container with id 4e10960e23c7968f3aca3eb52c76cd6db4f2a2d50efe81f579c53ecf629969b5 Sep 16 01:39:36.752530 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:36.752510 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx"] Sep 16 01:39:36.764421 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:39:36.764388 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9a500d_18fd_4b1f_9c9a_0c7a2d824a60.slice/crio-818e4555390e0e7700af70be81c7a04ca823b5aae1068b2bc773aefd081ecf85 WatchSource:0}: Error finding container 818e4555390e0e7700af70be81c7a04ca823b5aae1068b2bc773aefd081ecf85: Status 404 returned error can't find the container with id 818e4555390e0e7700af70be81c7a04ca823b5aae1068b2bc773aefd081ecf85 Sep 16 01:39:37.132868 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:37.132838 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:37.403597 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:37.403518 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" event={"ID":"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60","Type":"ContainerStarted","Data":"1e5ea52b8f27575c1c7882320835b25d3fce444563036ec136fbc9a394fbe71d"} Sep 16 01:39:37.403597 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:37.403574 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" event={"ID":"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60","Type":"ContainerStarted","Data":"818e4555390e0e7700af70be81c7a04ca823b5aae1068b2bc773aefd081ecf85"} Sep 16 01:39:37.403865 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:37.403722 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" Sep 16 01:39:37.405101 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:37.405080 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" event={"ID":"bbb68eb0-f56d-4538-9f8a-16eced43811b","Type":"ContainerStarted","Data":"a9e2606bce057626e346de0b1e27460443629e570da75545c233d7972d0e50b6"} Sep 16 01:39:37.405150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:37.405124 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" event={"ID":"bbb68eb0-f56d-4538-9f8a-16eced43811b","Type":"ContainerStarted","Data":"4e10960e23c7968f3aca3eb52c76cd6db4f2a2d50efe81f579c53ecf629969b5"} Sep 16 01:39:37.405207 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:37.405153 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" Sep 16 01:39:37.420346 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:37.420274 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" podStartSLOduration=1.420258607 podStartE2EDuration="1.420258607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:39:37.419715371 +0000 UTC m=+1407.672137988" watchObservedRunningTime="2026-09-16 01:39:37.420258607 +0000 UTC m=+1407.672681199" Sep 16 01:39:37.435829 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:37.435784 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" podStartSLOduration=1.4357706270000001 podStartE2EDuration="1.435770627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:39:37.432785614 +0000 UTC m=+1407.685208204" watchObservedRunningTime="2026-09-16 01:39:37.435770627 +0000 UTC m=+1407.688193217" Sep 16 01:39:37.870825 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:37.870796 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:48.412674 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:48.412640 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" Sep 16 01:39:48.413061 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:48.412702 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" Sep 16 01:39:51.967217 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:51.967147 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:52.067219 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:52.067181 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:53.312115 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:53.312086 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:39:57.549523 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:57.549494 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:58.136232 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:58.136194 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:58.236633 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:58.236608 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:58.869646 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:58.869618 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:39:59.477551 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:39:59.477512 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:00.322456 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:00.322388 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:09.216103 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.216020 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr"] Sep 16 01:40:09.219660 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.219639 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.222760 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.222728 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3b6e6489\"/\"openshift-service-ca.crt\"" Sep 16 01:40:09.222992 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.222971 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3b6e6489\"/\"default-dockercfg-vzr9n\"" Sep 16 01:40:09.222992 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.222985 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3b6e6489\"/\"kube-root-ca.crt\"" Sep 16 01:40:09.223185 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.222993 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3b6e6489\"/\"e2e-lifecycle-model-3b6e6489-kserve-self-signed-certs\"" Sep 16 01:40:09.231710 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.231686 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr"] Sep 16 01:40:09.233757 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.233733 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:09.357248 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.357211 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vfzk\" (UniqueName: \"kubernetes.io/projected/a810c823-cf3b-4b03-816d-cdb6650bc461-kube-api-access-2vfzk\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.357438 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.357329 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-tmp-dir\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.357438 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.357355 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-dshm\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.357535 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.357439 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-model-cache\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.357582 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.357536 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-home\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.357582 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.357560 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a810c823-cf3b-4b03-816d-cdb6650bc461-tls-certs\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.458852 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.458823 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vfzk\" (UniqueName: \"kubernetes.io/projected/a810c823-cf3b-4b03-816d-cdb6650bc461-kube-api-access-2vfzk\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.458990 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.458874 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-tmp-dir\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.459063 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.459040 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-dshm\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.459127 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.459111 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-model-cache\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.459222 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.459208 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-home\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.459284 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.459236 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a810c823-cf3b-4b03-816d-cdb6650bc461-tls-certs\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.459339 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.459281 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-tmp-dir\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.459607 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.459588 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-home\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.459758 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.459663 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-model-cache\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.461660 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.461640 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-dshm\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.462110 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.462092 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a810c823-cf3b-4b03-816d-cdb6650bc461-tls-certs\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.466014 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.465993 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vfzk\" (UniqueName: \"kubernetes.io/projected/a810c823-cf3b-4b03-816d-cdb6650bc461-kube-api-access-2vfzk\") pod \"e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.531218 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.531117 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:09.813206 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:09.813175 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr"] Sep 16 01:40:10.458464 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:10.458433 2558 scope.go:117] "RemoveContainer" containerID="12c9836f3177a2cb811afac3ae9ae63d6e68576d985e7fa6a1772432129759df" Sep 16 01:40:10.609652 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:10.609613 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" event={"ID":"a810c823-cf3b-4b03-816d-cdb6650bc461","Type":"ContainerStarted","Data":"54ff73f171b611009781e78447218bc1136f5329e0c51a744fd79af7a12c2b44"} Sep 16 01:40:10.609794 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:10.609664 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" event={"ID":"a810c823-cf3b-4b03-816d-cdb6650bc461","Type":"ContainerStarted","Data":"5c57406763d05133f879e08680902c222b764f83d1fe86b5db363dfed9ac969b"} Sep 16 01:40:10.616797 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:10.616773 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:40:10.624827 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:10.624761 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" podStartSLOduration=1.624747629 podStartE2EDuration="1.624747629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:40:10.624417375 +0000 UTC m=+1440.876839965" watchObservedRunningTime="2026-09-16 01:40:10.624747629 +0000 UTC m=+1440.877170217" Sep 16 01:40:19.531317 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:19.531276 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:19.531769 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:19.531340 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:19.543830 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:19.543806 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:19.679944 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:19.679916 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:40:26.550413 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:26.550381 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:26.656643 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:26.656613 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:27.521892 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:27.521859 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:40:27.627180 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:27.626224 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:27.632512 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:27.632481 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw"] Sep 16 01:40:34.031012 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:34.030983 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:38.186876 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:38.186842 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:39.295756 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:39.295728 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:39.822405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:39.822372 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:40:39.969474 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:39.969446 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:40.879520 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:40.879493 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:41.493321 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:41.493287 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:42.268959 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.268926 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:42.654123 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.654081 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" podUID="88ce1004-8ab1-415f-9be0-c8ea53e54a75" containerName="main" containerID="cri-o://a131a0788fc8c3262630065cf4f0d9309ef57abbbd51c132edc793ed53753498" gracePeriod=45 Sep 16 01:40:42.810624 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.810587 2558 generic.go:356] "Generic (PLEG): container finished" podID="88ce1004-8ab1-415f-9be0-c8ea53e54a75" containerID="a131a0788fc8c3262630065cf4f0d9309ef57abbbd51c132edc793ed53753498" exitCode=0 Sep 16 01:40:42.810786 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.810656 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" event={"ID":"88ce1004-8ab1-415f-9be0-c8ea53e54a75","Type":"ContainerDied","Data":"a131a0788fc8c3262630065cf4f0d9309ef57abbbd51c132edc793ed53753498"} Sep 16 01:40:42.916741 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.916679 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:40:42.964327 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.964300 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfzg7\" (UniqueName: \"kubernetes.io/projected/88ce1004-8ab1-415f-9be0-c8ea53e54a75-kube-api-access-bfzg7\") pod \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " Sep 16 01:40:42.964496 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.964342 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-model-cache\") pod \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " Sep 16 01:40:42.964496 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.964396 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-home\") pod \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " Sep 16 01:40:42.964496 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.964430 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-dshm\") pod \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " Sep 16 01:40:42.964496 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.964475 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88ce1004-8ab1-415f-9be0-c8ea53e54a75-tls-certs\") pod \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " Sep 16 01:40:42.964754 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.964497 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-tmp-dir\") pod \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\" (UID: \"88ce1004-8ab1-415f-9be0-c8ea53e54a75\") " Sep 16 01:40:42.964754 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.964599 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-model-cache" (OuterVolumeSpecName: "model-cache") pod "88ce1004-8ab1-415f-9be0-c8ea53e54a75" (UID: "88ce1004-8ab1-415f-9be0-c8ea53e54a75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:40:42.964754 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.964685 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-home" (OuterVolumeSpecName: "home") pod "88ce1004-8ab1-415f-9be0-c8ea53e54a75" (UID: "88ce1004-8ab1-415f-9be0-c8ea53e54a75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:40:42.964922 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.964867 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-model-cache\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:40:42.964922 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.964886 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-home\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:40:42.965225 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.965065 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88ce1004-8ab1-415f-9be0-c8ea53e54a75" (UID: "88ce1004-8ab1-415f-9be0-c8ea53e54a75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:40:42.967011 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.966987 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-dshm" (OuterVolumeSpecName: "dshm") pod "88ce1004-8ab1-415f-9be0-c8ea53e54a75" (UID: "88ce1004-8ab1-415f-9be0-c8ea53e54a75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:40:42.967011 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.967003 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ce1004-8ab1-415f-9be0-c8ea53e54a75-kube-api-access-bfzg7" (OuterVolumeSpecName: "kube-api-access-bfzg7") pod "88ce1004-8ab1-415f-9be0-c8ea53e54a75" (UID: "88ce1004-8ab1-415f-9be0-c8ea53e54a75"). InnerVolumeSpecName "kube-api-access-bfzg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:40:42.967210 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:42.967189 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88ce1004-8ab1-415f-9be0-c8ea53e54a75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88ce1004-8ab1-415f-9be0-c8ea53e54a75" (UID: "88ce1004-8ab1-415f-9be0-c8ea53e54a75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:40:43.066390 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:43.066355 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88ce1004-8ab1-415f-9be0-c8ea53e54a75-tls-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:40:43.066516 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:43.066415 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-tmp-dir\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:40:43.066516 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:43.066440 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfzg7\" (UniqueName: \"kubernetes.io/projected/88ce1004-8ab1-415f-9be0-c8ea53e54a75-kube-api-access-bfzg7\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:40:43.066516 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:43.066454 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88ce1004-8ab1-415f-9be0-c8ea53e54a75-dshm\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:40:43.817917 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:43.817889 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" Sep 16 01:40:43.818421 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:43.817890 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" event={"ID":"88ce1004-8ab1-415f-9be0-c8ea53e54a75","Type":"ContainerDied","Data":"d79cdfaf5d607c01affc6894b9cabd8ac9bfe96a7b4f3a07559730db98022e95"} Sep 16 01:40:43.818421 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:43.818019 2558 scope.go:117] "RemoveContainer" containerID="a131a0788fc8c3262630065cf4f0d9309ef57abbbd51c132edc793ed53753498" Sep 16 01:40:43.839672 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:43.839649 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw"] Sep 16 01:40:43.842036 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:43.842016 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw"] Sep 16 01:40:44.295900 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:44.295870 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88ce1004-8ab1-415f-9be0-c8ea53e54a75" path="/var/lib/kubelet/pods/88ce1004-8ab1-415f-9be0-c8ea53e54a75/volumes" Sep 16 01:40:47.861177 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:47.861118 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-c0ec73/auth-test-model-c0ec73-kserve-68899c5759-t2jzw" podUID="88ce1004-8ab1-415f-9be0-c8ea53e54a75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Sep 16 01:40:51.517604 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.517569 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc"] Sep 16 01:40:51.518522 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.518227 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ce1004-8ab1-415f-9be0-c8ea53e54a75" containerName="main" Sep 16 01:40:51.518522 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.518247 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ce1004-8ab1-415f-9be0-c8ea53e54a75" containerName="main" Sep 16 01:40:51.518522 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.518405 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ce1004-8ab1-415f-9be0-c8ea53e54a75" containerName="main" Sep 16 01:40:51.523139 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.523114 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.526629 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.526603 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c0ec73\"/\"openshift-service-ca.crt\"" Sep 16 01:40:51.526868 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.526618 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c0ec73\"/\"default-dockercfg-p4jqz\"" Sep 16 01:40:51.526974 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.526667 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c0ec73\"/\"sub-test-model-c0ec73-kserve-self-signed-certs\"" Sep 16 01:40:51.527036 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.526673 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c0ec73\"/\"kube-root-ca.crt\"" Sep 16 01:40:51.531483 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.531457 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc"] Sep 16 01:40:51.535385 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.535323 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:51.646469 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.646433 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-home\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.646469 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.646478 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-tmp-dir\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.646655 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.646500 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nc7c\" (UniqueName: \"kubernetes.io/projected/1b596e9e-6243-4e33-ad5c-a066d92903f8-kube-api-access-2nc7c\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.646655 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.646591 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-model-cache\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.646655 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.646643 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b596e9e-6243-4e33-ad5c-a066d92903f8-tls-certs\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.646759 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.646698 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-dshm\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.747776 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.747737 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nc7c\" (UniqueName: \"kubernetes.io/projected/1b596e9e-6243-4e33-ad5c-a066d92903f8-kube-api-access-2nc7c\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.747942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.747793 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-model-cache\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.747942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.747840 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b596e9e-6243-4e33-ad5c-a066d92903f8-tls-certs\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.747942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.747869 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-dshm\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.747942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.747938 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-home\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.748121 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.747973 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-tmp-dir\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.748404 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.748378 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-model-cache\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.748404 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.748396 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-tmp-dir\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.748528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.748418 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-home\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.750385 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.750363 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-dshm\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.750716 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.750657 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b596e9e-6243-4e33-ad5c-a066d92903f8-tls-certs\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.754890 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.754867 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nc7c\" (UniqueName: \"kubernetes.io/projected/1b596e9e-6243-4e33-ad5c-a066d92903f8-kube-api-access-2nc7c\") pod \"sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:51.838785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:51.838715 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:40:52.113010 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:52.112983 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc"] Sep 16 01:40:52.115131 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:40:52.115104 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b596e9e_6243_4e33_ad5c_a066d92903f8.slice/crio-f0d9119a90c65eb64c44b01acfa87c6b4e4b27025087491723572570749684f2 WatchSource:0}: Error finding container f0d9119a90c65eb64c44b01acfa87c6b4e4b27025087491723572570749684f2: Status 404 returned error can't find the container with id f0d9119a90c65eb64c44b01acfa87c6b4e4b27025087491723572570749684f2 Sep 16 01:40:52.878049 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:52.878015 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" event={"ID":"1b596e9e-6243-4e33-ad5c-a066d92903f8","Type":"ContainerStarted","Data":"88bf458bdc7507c69de011477fb24ecb556a55b6954b483a9c321735caacd483"} Sep 16 01:40:52.878441 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:52.878056 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" event={"ID":"1b596e9e-6243-4e33-ad5c-a066d92903f8","Type":"ContainerStarted","Data":"f0d9119a90c65eb64c44b01acfa87c6b4e4b27025087491723572570749684f2"} Sep 16 01:40:52.898450 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:52.898409 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" podStartSLOduration=1.898395118 podStartE2EDuration="1.898395118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:40:52.897358217 +0000 UTC m=+1483.149780819" watchObservedRunningTime="2026-09-16 01:40:52.898395118 +0000 UTC m=+1483.150817707" Sep 16 01:40:53.312109 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.312014 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:40:53.555385 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.555334 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:53.581597 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.581516 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx"] Sep 16 01:40:53.581814 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.581790 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" podUID="3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60" containerName="payload-processing" containerID="cri-o://1e5ea52b8f27575c1c7882320835b25d3fce444563036ec136fbc9a394fbe71d" gracePeriod=30 Sep 16 01:40:53.587295 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.586949 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr"] Sep 16 01:40:53.587433 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.587293 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" podUID="bbb68eb0-f56d-4538-9f8a-16eced43811b" containerName="payload-pre-processing" containerID="cri-o://a9e2606bce057626e346de0b1e27460443629e570da75545c233d7972d0e50b6" gracePeriod=30 Sep 16 01:40:53.618785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.618755 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:53.676041 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.675981 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:53.732004 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.731972 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:53.785546 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.785432 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:53.871351 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.871329 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" Sep 16 01:40:53.874414 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.874397 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" Sep 16 01:40:53.885344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.885314 2558 generic.go:356] "Generic (PLEG): container finished" podID="3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60" containerID="1e5ea52b8f27575c1c7882320835b25d3fce444563036ec136fbc9a394fbe71d" exitCode=0 Sep 16 01:40:53.885693 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.885380 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" Sep 16 01:40:53.885693 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.885407 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" event={"ID":"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60","Type":"ContainerDied","Data":"1e5ea52b8f27575c1c7882320835b25d3fce444563036ec136fbc9a394fbe71d"} Sep 16 01:40:53.885693 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.885458 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx" event={"ID":"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60","Type":"ContainerDied","Data":"818e4555390e0e7700af70be81c7a04ca823b5aae1068b2bc773aefd081ecf85"} Sep 16 01:40:53.885693 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.885486 2558 scope.go:117] "RemoveContainer" containerID="1e5ea52b8f27575c1c7882320835b25d3fce444563036ec136fbc9a394fbe71d" Sep 16 01:40:53.888028 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.888003 2558 generic.go:356] "Generic (PLEG): container finished" podID="bbb68eb0-f56d-4538-9f8a-16eced43811b" containerID="a9e2606bce057626e346de0b1e27460443629e570da75545c233d7972d0e50b6" exitCode=0 Sep 16 01:40:53.888146 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.888047 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" Sep 16 01:40:53.888146 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.888099 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" event={"ID":"bbb68eb0-f56d-4538-9f8a-16eced43811b","Type":"ContainerDied","Data":"a9e2606bce057626e346de0b1e27460443629e570da75545c233d7972d0e50b6"} Sep 16 01:40:53.888294 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.888177 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr" event={"ID":"bbb68eb0-f56d-4538-9f8a-16eced43811b","Type":"ContainerDied","Data":"4e10960e23c7968f3aca3eb52c76cd6db4f2a2d50efe81f579c53ecf629969b5"} Sep 16 01:40:53.896297 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.896278 2558 scope.go:117] "RemoveContainer" containerID="1e5ea52b8f27575c1c7882320835b25d3fce444563036ec136fbc9a394fbe71d" Sep 16 01:40:53.896592 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:40:53.896565 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5ea52b8f27575c1c7882320835b25d3fce444563036ec136fbc9a394fbe71d\": container with ID starting with 1e5ea52b8f27575c1c7882320835b25d3fce444563036ec136fbc9a394fbe71d not found: ID does not exist" containerID="1e5ea52b8f27575c1c7882320835b25d3fce444563036ec136fbc9a394fbe71d" Sep 16 01:40:53.896655 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.896605 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5ea52b8f27575c1c7882320835b25d3fce444563036ec136fbc9a394fbe71d"} err="failed to get container status \"1e5ea52b8f27575c1c7882320835b25d3fce444563036ec136fbc9a394fbe71d\": rpc error: code = NotFound desc = could not find container \"1e5ea52b8f27575c1c7882320835b25d3fce444563036ec136fbc9a394fbe71d\": container with ID starting with 1e5ea52b8f27575c1c7882320835b25d3fce444563036ec136fbc9a394fbe71d not found: ID does not exist" Sep 16 01:40:53.896655 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.896630 2558 scope.go:117] "RemoveContainer" containerID="a9e2606bce057626e346de0b1e27460443629e570da75545c233d7972d0e50b6" Sep 16 01:40:53.907236 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.907125 2558 scope.go:117] "RemoveContainer" containerID="a9e2606bce057626e346de0b1e27460443629e570da75545c233d7972d0e50b6" Sep 16 01:40:53.907618 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:40:53.907514 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e2606bce057626e346de0b1e27460443629e570da75545c233d7972d0e50b6\": container with ID starting with a9e2606bce057626e346de0b1e27460443629e570da75545c233d7972d0e50b6 not found: ID does not exist" containerID="a9e2606bce057626e346de0b1e27460443629e570da75545c233d7972d0e50b6" Sep 16 01:40:53.907618 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.907549 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e2606bce057626e346de0b1e27460443629e570da75545c233d7972d0e50b6"} err="failed to get container status \"a9e2606bce057626e346de0b1e27460443629e570da75545c233d7972d0e50b6\": rpc error: code = NotFound desc = could not find container \"a9e2606bce057626e346de0b1e27460443629e570da75545c233d7972d0e50b6\": container with ID starting with a9e2606bce057626e346de0b1e27460443629e570da75545c233d7972d0e50b6 not found: ID does not exist" Sep 16 01:40:53.967747 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.967721 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60-plugins-config-volume\") pod \"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60\" (UID: \"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60\") " Sep 16 01:40:53.967901 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.967780 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb68eb0-f56d-4538-9f8a-16eced43811b-plugins-config-volume\") pod \"bbb68eb0-f56d-4538-9f8a-16eced43811b\" (UID: \"bbb68eb0-f56d-4538-9f8a-16eced43811b\") " Sep 16 01:40:53.967987 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.967967 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rrq2\" (UniqueName: \"kubernetes.io/projected/bbb68eb0-f56d-4538-9f8a-16eced43811b-kube-api-access-6rrq2\") pod \"bbb68eb0-f56d-4538-9f8a-16eced43811b\" (UID: \"bbb68eb0-f56d-4538-9f8a-16eced43811b\") " Sep 16 01:40:53.968055 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.967999 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv8vl\" (UniqueName: \"kubernetes.io/projected/3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60-kube-api-access-rv8vl\") pod \"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60\" (UID: \"3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60\") " Sep 16 01:40:53.968117 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.968066 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60" (UID: "3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:40:53.968228 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.968208 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb68eb0-f56d-4538-9f8a-16eced43811b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbb68eb0-f56d-4538-9f8a-16eced43811b" (UID: "bbb68eb0-f56d-4538-9f8a-16eced43811b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:40:53.968421 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.968404 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:40:53.968480 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.968427 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb68eb0-f56d-4538-9f8a-16eced43811b-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:40:53.970266 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.970243 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb68eb0-f56d-4538-9f8a-16eced43811b-kube-api-access-6rrq2" (OuterVolumeSpecName: "kube-api-access-6rrq2") pod "bbb68eb0-f56d-4538-9f8a-16eced43811b" (UID: "bbb68eb0-f56d-4538-9f8a-16eced43811b"). InnerVolumeSpecName "kube-api-access-6rrq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:40:53.970355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:53.970297 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60-kube-api-access-rv8vl" (OuterVolumeSpecName: "kube-api-access-rv8vl") pod "3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60" (UID: "3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60"). InnerVolumeSpecName "kube-api-access-rv8vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:40:54.069392 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:54.069355 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rrq2\" (UniqueName: \"kubernetes.io/projected/bbb68eb0-f56d-4538-9f8a-16eced43811b-kube-api-access-6rrq2\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:40:54.069392 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:54.069385 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv8vl\" (UniqueName: \"kubernetes.io/projected/3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60-kube-api-access-rv8vl\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:40:54.220005 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:54.219971 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx"] Sep 16 01:40:54.227951 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:54.227921 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3b6e6489-6d9c69865-swtqx"] Sep 16 01:40:54.245699 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:54.245673 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr"] Sep 16 01:40:54.261114 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:54.261084 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3b6e6489-85cd45c59d-7gwsr"] Sep 16 01:40:54.296596 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:54.296566 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60" path="/var/lib/kubelet/pods/3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60/volumes" Sep 16 01:40:54.296995 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:54.296974 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbb68eb0-f56d-4538-9f8a-16eced43811b" path="/var/lib/kubelet/pods/bbb68eb0-f56d-4538-9f8a-16eced43811b/volumes" Sep 16 01:40:54.320212 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:54.320179 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:55.067995 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:55.067959 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:55.669747 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:55.669717 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:56.368872 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:56.368839 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:40:57.105501 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:40:57.105471 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:01.838833 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:01.838793 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:41:01.839283 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:01.838853 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:41:01.851511 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:01.851490 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:41:01.951477 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:01.951447 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:41:02.093970 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:02.093888 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr"] Sep 16 01:41:02.130820 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:02.130787 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:04.895405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:04.895374 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:06.212279 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:06.212241 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:41:10.340243 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:10.340215 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:41:10.344617 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:10.344593 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:41:10.502854 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:10.502827 2558 scope.go:117] "RemoveContainer" containerID="7c46e49aec43e51ed028a085fee494de1ee940e1164d9bb4b4020984236fa606" Sep 16 01:41:14.623904 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:14.623863 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:41:17.114033 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.113987 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" podUID="a810c823-cf3b-4b03-816d-cdb6650bc461" containerName="main" containerID="cri-o://54ff73f171b611009781e78447218bc1136f5329e0c51a744fd79af7a12c2b44" gracePeriod=45 Sep 16 01:41:17.379634 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.379565 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:41:17.476925 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.476881 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-dshm\") pod \"a810c823-cf3b-4b03-816d-cdb6650bc461\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " Sep 16 01:41:17.477102 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.476939 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vfzk\" (UniqueName: \"kubernetes.io/projected/a810c823-cf3b-4b03-816d-cdb6650bc461-kube-api-access-2vfzk\") pod \"a810c823-cf3b-4b03-816d-cdb6650bc461\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " Sep 16 01:41:17.477102 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.476996 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-model-cache\") pod \"a810c823-cf3b-4b03-816d-cdb6650bc461\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " Sep 16 01:41:17.477102 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.477052 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-tmp-dir\") pod \"a810c823-cf3b-4b03-816d-cdb6650bc461\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " Sep 16 01:41:17.477102 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.477082 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-home\") pod \"a810c823-cf3b-4b03-816d-cdb6650bc461\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " Sep 16 01:41:17.477345 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.477195 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a810c823-cf3b-4b03-816d-cdb6650bc461-tls-certs\") pod \"a810c823-cf3b-4b03-816d-cdb6650bc461\" (UID: \"a810c823-cf3b-4b03-816d-cdb6650bc461\") " Sep 16 01:41:17.477724 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.477403 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a810c823-cf3b-4b03-816d-cdb6650bc461" (UID: "a810c823-cf3b-4b03-816d-cdb6650bc461"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:41:17.477724 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.477627 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-home" (OuterVolumeSpecName: "home") pod "a810c823-cf3b-4b03-816d-cdb6650bc461" (UID: "a810c823-cf3b-4b03-816d-cdb6650bc461"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:41:17.477724 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.477672 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-tmp-dir\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:17.477931 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.477824 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-model-cache" (OuterVolumeSpecName: "model-cache") pod "a810c823-cf3b-4b03-816d-cdb6650bc461" (UID: "a810c823-cf3b-4b03-816d-cdb6650bc461"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:41:17.479589 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.479564 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-dshm" (OuterVolumeSpecName: "dshm") pod "a810c823-cf3b-4b03-816d-cdb6650bc461" (UID: "a810c823-cf3b-4b03-816d-cdb6650bc461"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:41:17.479673 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.479591 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a810c823-cf3b-4b03-816d-cdb6650bc461-kube-api-access-2vfzk" (OuterVolumeSpecName: "kube-api-access-2vfzk") pod "a810c823-cf3b-4b03-816d-cdb6650bc461" (UID: "a810c823-cf3b-4b03-816d-cdb6650bc461"). InnerVolumeSpecName "kube-api-access-2vfzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:41:17.479715 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.479699 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a810c823-cf3b-4b03-816d-cdb6650bc461-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a810c823-cf3b-4b03-816d-cdb6650bc461" (UID: "a810c823-cf3b-4b03-816d-cdb6650bc461"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:41:17.578879 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.578856 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-dshm\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:17.578879 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.578878 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vfzk\" (UniqueName: \"kubernetes.io/projected/a810c823-cf3b-4b03-816d-cdb6650bc461-kube-api-access-2vfzk\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:17.578879 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.578888 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-model-cache\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:17.578879 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.578897 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a810c823-cf3b-4b03-816d-cdb6650bc461-home\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:17.579128 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:17.578904 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a810c823-cf3b-4b03-816d-cdb6650bc461-tls-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:18.038137 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:18.038106 2558 generic.go:356] "Generic (PLEG): container finished" podID="a810c823-cf3b-4b03-816d-cdb6650bc461" containerID="54ff73f171b611009781e78447218bc1136f5329e0c51a744fd79af7a12c2b44" exitCode=0 Sep 16 01:41:18.038315 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:18.038145 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" event={"ID":"a810c823-cf3b-4b03-816d-cdb6650bc461","Type":"ContainerDied","Data":"54ff73f171b611009781e78447218bc1136f5329e0c51a744fd79af7a12c2b44"} Sep 16 01:41:18.038315 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:18.038150 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" Sep 16 01:41:18.038315 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:18.038205 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr" event={"ID":"a810c823-cf3b-4b03-816d-cdb6650bc461","Type":"ContainerDied","Data":"5c57406763d05133f879e08680902c222b764f83d1fe86b5db363dfed9ac969b"} Sep 16 01:41:18.038315 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:18.038223 2558 scope.go:117] "RemoveContainer" containerID="54ff73f171b611009781e78447218bc1136f5329e0c51a744fd79af7a12c2b44" Sep 16 01:41:18.048655 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:18.048638 2558 scope.go:117] "RemoveContainer" containerID="54ff73f171b611009781e78447218bc1136f5329e0c51a744fd79af7a12c2b44" Sep 16 01:41:18.048918 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:41:18.048896 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ff73f171b611009781e78447218bc1136f5329e0c51a744fd79af7a12c2b44\": container with ID starting with 54ff73f171b611009781e78447218bc1136f5329e0c51a744fd79af7a12c2b44 not found: ID does not exist" containerID="54ff73f171b611009781e78447218bc1136f5329e0c51a744fd79af7a12c2b44" Sep 16 01:41:18.048975 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:18.048928 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ff73f171b611009781e78447218bc1136f5329e0c51a744fd79af7a12c2b44"} err="failed to get container status \"54ff73f171b611009781e78447218bc1136f5329e0c51a744fd79af7a12c2b44\": rpc error: code = NotFound desc = could not find container \"54ff73f171b611009781e78447218bc1136f5329e0c51a744fd79af7a12c2b44\": container with ID starting with 54ff73f171b611009781e78447218bc1136f5329e0c51a744fd79af7a12c2b44 not found: ID does not exist" Sep 16 01:41:18.060410 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:18.060385 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr"] Sep 16 01:41:18.064405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:18.064383 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3b6e6489/e2e-lifecycle-model-3b6e6489-kserve-6f55dd6787-mtdxr"] Sep 16 01:41:18.295662 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:18.295575 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a810c823-cf3b-4b03-816d-cdb6650bc461" path="/var/lib/kubelet/pods/a810c823-cf3b-4b03-816d-cdb6650bc461/volumes" Sep 16 01:41:26.020399 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.020367 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9138-2dd0d0"] Sep 16 01:41:26.020845 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.020778 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a810c823-cf3b-4b03-816d-cdb6650bc461" containerName="main" Sep 16 01:41:26.020845 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.020789 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a810c823-cf3b-4b03-816d-cdb6650bc461" containerName="main" Sep 16 01:41:26.020845 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.020815 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60" containerName="payload-processing" Sep 16 01:41:26.020845 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.020821 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60" containerName="payload-processing" Sep 16 01:41:26.020845 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.020829 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbb68eb0-f56d-4538-9f8a-16eced43811b" containerName="payload-pre-processing" Sep 16 01:41:26.020845 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.020835 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbb68eb0-f56d-4538-9f8a-16eced43811b" containerName="payload-pre-processing" Sep 16 01:41:26.021043 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.020895 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a9a500d-18fd-4b1f-9c9a-0c7a2d824a60" containerName="payload-processing" Sep 16 01:41:26.021043 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.020903 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a810c823-cf3b-4b03-816d-cdb6650bc461" containerName="main" Sep 16 01:41:26.021043 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.020910 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbb68eb0-f56d-4538-9f8a-16eced43811b" containerName="payload-pre-processing" Sep 16 01:41:26.025736 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.025698 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9138-2dd0d0" Sep 16 01:41:26.027809 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.027784 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6n2fb\"" Sep 16 01:41:26.028571 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.028549 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9138-2dd0d0"] Sep 16 01:41:26.052643 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.052617 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdxkt\" (UniqueName: \"kubernetes.io/projected/a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c-kube-api-access-cdxkt\") pod \"test-curl-9138-2dd0d0\" (UID: \"a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c\") " pod="opendatahub/test-curl-9138-2dd0d0" Sep 16 01:41:26.153409 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.153379 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdxkt\" (UniqueName: \"kubernetes.io/projected/a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c-kube-api-access-cdxkt\") pod \"test-curl-9138-2dd0d0\" (UID: \"a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c\") " pod="opendatahub/test-curl-9138-2dd0d0" Sep 16 01:41:26.161681 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.161656 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdxkt\" (UniqueName: \"kubernetes.io/projected/a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c-kube-api-access-cdxkt\") pod \"test-curl-9138-2dd0d0\" (UID: \"a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c\") " pod="opendatahub/test-curl-9138-2dd0d0" Sep 16 01:41:26.336513 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.336435 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9138-2dd0d0" Sep 16 01:41:26.609025 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:26.608998 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9138-2dd0d0"] Sep 16 01:41:26.610346 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:41:26.610319 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3f3b9dd_4490_40d3_9da6_4a7dde8ad12c.slice/crio-f78d51eefe7b424d781138b199cd9555987d58b236effa68ae8f9c69cbc8f298 WatchSource:0}: Error finding container f78d51eefe7b424d781138b199cd9555987d58b236effa68ae8f9c69cbc8f298: Status 404 returned error can't find the container with id f78d51eefe7b424d781138b199cd9555987d58b236effa68ae8f9c69cbc8f298 Sep 16 01:41:27.098219 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.098187 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9138-2dd0d0" event={"ID":"a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c","Type":"ContainerStarted","Data":"e9422074c4ac7ada6a9375af9974ca8dc7c0f4d47954b42e28cb977ca0f16994"} Sep 16 01:41:27.098219 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.098230 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9138-2dd0d0" event={"ID":"a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c","Type":"ContainerStarted","Data":"f78d51eefe7b424d781138b199cd9555987d58b236effa68ae8f9c69cbc8f298"} Sep 16 01:41:27.113303 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.113253 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9138-2dd0d0" podStartSLOduration=1.113239244 podStartE2EDuration="1.113239244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:41:27.109974831 +0000 UTC m=+1517.362397420" watchObservedRunningTime="2026-09-16 01:41:27.113239244 +0000 UTC m=+1517.365661833" Sep 16 01:41:27.710181 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.710135 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9138-2dd0d0"] Sep 16 01:41:27.712837 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.712811 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9138-2dd0d0"] Sep 16 01:41:27.830942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.830872 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9138-5c3300"] Sep 16 01:41:27.831338 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.831323 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c" containerName="test-curl-9138-2dd0d0" Sep 16 01:41:27.831390 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.831340 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c" containerName="test-curl-9138-2dd0d0" Sep 16 01:41:27.831430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.831412 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c" containerName="test-curl-9138-2dd0d0" Sep 16 01:41:27.834611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.834595 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9138-5c3300" Sep 16 01:41:27.839647 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.839624 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9138-5c3300"] Sep 16 01:41:27.866763 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.866740 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49zf\" (UniqueName: \"kubernetes.io/projected/3c217985-0d86-4823-b953-41d9e4854290-kube-api-access-d49zf\") pod \"test-curl-9138-5c3300\" (UID: \"3c217985-0d86-4823-b953-41d9e4854290\") " pod="opendatahub/test-curl-9138-5c3300" Sep 16 01:41:27.967553 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.967520 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49zf\" (UniqueName: \"kubernetes.io/projected/3c217985-0d86-4823-b953-41d9e4854290-kube-api-access-d49zf\") pod \"test-curl-9138-5c3300\" (UID: \"3c217985-0d86-4823-b953-41d9e4854290\") " pod="opendatahub/test-curl-9138-5c3300" Sep 16 01:41:27.974434 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:27.974404 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49zf\" (UniqueName: \"kubernetes.io/projected/3c217985-0d86-4823-b953-41d9e4854290-kube-api-access-d49zf\") pod \"test-curl-9138-5c3300\" (UID: \"3c217985-0d86-4823-b953-41d9e4854290\") " pod="opendatahub/test-curl-9138-5c3300" Sep 16 01:41:28.104669 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:41:28.104597 2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9138-2dd0d0" container="test-curl-9138-2dd0d0" Sep 16 01:41:28.146965 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:28.146940 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9138-5c3300" Sep 16 01:41:28.415219 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:28.415194 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9138-5c3300"] Sep 16 01:41:28.416986 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:41:28.416965 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c217985_0d86_4823_b953_41d9e4854290.slice/crio-9928c304b66c78a46ac8b517cd4c4b68e73f691dda4b6def221c64e6b9d41481 WatchSource:0}: Error finding container 9928c304b66c78a46ac8b517cd4c4b68e73f691dda4b6def221c64e6b9d41481: Status 404 returned error can't find the container with id 9928c304b66c78a46ac8b517cd4c4b68e73f691dda4b6def221c64e6b9d41481 Sep 16 01:41:29.111459 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:29.111421 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9138-5c3300" event={"ID":"3c217985-0d86-4823-b953-41d9e4854290","Type":"ContainerStarted","Data":"55782ea1b697125425ec0e1ecf5b969c76aa29521da26c5850c9d759a59821cf"} Sep 16 01:41:29.111459 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:29.111461 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9138-5c3300" event={"ID":"3c217985-0d86-4823-b953-41d9e4854290","Type":"ContainerStarted","Data":"9928c304b66c78a46ac8b517cd4c4b68e73f691dda4b6def221c64e6b9d41481"} Sep 16 01:41:29.111944 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:29.111554 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9138-2dd0d0" podUID="a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c" containerName="test-curl-9138-2dd0d0" containerID="cri-o://e9422074c4ac7ada6a9375af9974ca8dc7c0f4d47954b42e28cb977ca0f16994" gracePeriod=2 Sep 16 01:41:29.126074 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:29.126021 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9138-5c3300" podStartSLOduration=2.126007367 podStartE2EDuration="2.126007367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:41:29.123214194 +0000 UTC m=+1519.375636783" watchObservedRunningTime="2026-09-16 01:41:29.126007367 +0000 UTC m=+1519.378429955" Sep 16 01:41:29.708436 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:29.708399 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9138-5c3300"] Sep 16 01:41:29.712344 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:29.712314 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9138-5c3300"] Sep 16 01:41:30.116816 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:41:30.116784 2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9138-5c3300" container="test-curl-9138-5c3300" Sep 16 01:41:31.122541 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:31.122510 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9138-5c3300" podUID="3c217985-0d86-4823-b953-41d9e4854290" containerName="test-curl-9138-5c3300" containerID="cri-o://55782ea1b697125425ec0e1ecf5b969c76aa29521da26c5850c9d759a59821cf" gracePeriod=2 Sep 16 01:41:31.294755 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:31.294732 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9138-2dd0d0" Sep 16 01:41:31.325541 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:31.325519 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:41:31.400379 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:31.400286 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdxkt\" (UniqueName: \"kubernetes.io/projected/a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c-kube-api-access-cdxkt\") pod \"a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c\" (UID: \"a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c\") " Sep 16 01:41:31.402479 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:31.402456 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c-kube-api-access-cdxkt" (OuterVolumeSpecName: "kube-api-access-cdxkt") pod "a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c" (UID: "a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c"). InnerVolumeSpecName "kube-api-access-cdxkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:41:31.501758 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:31.501731 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdxkt\" (UniqueName: \"kubernetes.io/projected/a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c-kube-api-access-cdxkt\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:32.130092 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:32.130056 2558 generic.go:356] "Generic (PLEG): container finished" podID="a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c" containerID="e9422074c4ac7ada6a9375af9974ca8dc7c0f4d47954b42e28cb977ca0f16994" exitCode=137 Sep 16 01:41:32.130507 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:32.130097 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9138-2dd0d0" Sep 16 01:41:32.130507 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:32.130179 2558 scope.go:117] "RemoveContainer" containerID="e9422074c4ac7ada6a9375af9974ca8dc7c0f4d47954b42e28cb977ca0f16994" Sep 16 01:41:32.140266 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:41:32.140236 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9138-2dd0d0\" is forbidden: User \"system:node:ip-10-0-137-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-55.ec2.internal' and this object" podUID="a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c" pod="opendatahub/test-curl-9138-2dd0d0" Sep 16 01:41:32.140444 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:32.140433 2558 scope.go:117] "RemoveContainer" containerID="e9422074c4ac7ada6a9375af9974ca8dc7c0f4d47954b42e28cb977ca0f16994" Sep 16 01:41:32.140696 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:41:32.140678 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9422074c4ac7ada6a9375af9974ca8dc7c0f4d47954b42e28cb977ca0f16994\": container with ID starting with e9422074c4ac7ada6a9375af9974ca8dc7c0f4d47954b42e28cb977ca0f16994 not found: ID does not exist" containerID="e9422074c4ac7ada6a9375af9974ca8dc7c0f4d47954b42e28cb977ca0f16994" Sep 16 01:41:32.140735 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:32.140704 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9422074c4ac7ada6a9375af9974ca8dc7c0f4d47954b42e28cb977ca0f16994"} err="failed to get container status \"e9422074c4ac7ada6a9375af9974ca8dc7c0f4d47954b42e28cb977ca0f16994\": rpc error: code = NotFound desc = could not find container \"e9422074c4ac7ada6a9375af9974ca8dc7c0f4d47954b42e28cb977ca0f16994\": container with ID starting with e9422074c4ac7ada6a9375af9974ca8dc7c0f4d47954b42e28cb977ca0f16994 not found: ID does not exist" Sep 16 01:41:32.295304 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:32.295266 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c" path="/var/lib/kubelet/pods/a3f3b9dd-4490-40d3-9da6-4a7dde8ad12c/volumes" Sep 16 01:41:32.528898 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:32.528819 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:32.820476 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:32.820396 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc"] Sep 16 01:41:32.838327 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:32.838296 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:33.289697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:33.289674 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9138-5c3300" Sep 16 01:41:33.316457 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:33.316437 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d49zf\" (UniqueName: \"kubernetes.io/projected/3c217985-0d86-4823-b953-41d9e4854290-kube-api-access-d49zf\") pod \"3c217985-0d86-4823-b953-41d9e4854290\" (UID: \"3c217985-0d86-4823-b953-41d9e4854290\") " Sep 16 01:41:33.318573 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:33.318551 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c217985-0d86-4823-b953-41d9e4854290-kube-api-access-d49zf" (OuterVolumeSpecName: "kube-api-access-d49zf") pod "3c217985-0d86-4823-b953-41d9e4854290" (UID: "3c217985-0d86-4823-b953-41d9e4854290"). InnerVolumeSpecName "kube-api-access-d49zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:41:33.417315 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:33.417255 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d49zf\" (UniqueName: \"kubernetes.io/projected/3c217985-0d86-4823-b953-41d9e4854290-kube-api-access-d49zf\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:34.144553 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:34.144519 2558 generic.go:356] "Generic (PLEG): container finished" podID="3c217985-0d86-4823-b953-41d9e4854290" containerID="55782ea1b697125425ec0e1ecf5b969c76aa29521da26c5850c9d759a59821cf" exitCode=137 Sep 16 01:41:34.144725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:34.144568 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9138-5c3300" Sep 16 01:41:34.144725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:34.144619 2558 scope.go:117] "RemoveContainer" containerID="55782ea1b697125425ec0e1ecf5b969c76aa29521da26c5850c9d759a59821cf" Sep 16 01:41:34.154023 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:34.153872 2558 scope.go:117] "RemoveContainer" containerID="55782ea1b697125425ec0e1ecf5b969c76aa29521da26c5850c9d759a59821cf" Sep 16 01:41:34.154185 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:41:34.154147 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55782ea1b697125425ec0e1ecf5b969c76aa29521da26c5850c9d759a59821cf\": container with ID starting with 55782ea1b697125425ec0e1ecf5b969c76aa29521da26c5850c9d759a59821cf not found: ID does not exist" containerID="55782ea1b697125425ec0e1ecf5b969c76aa29521da26c5850c9d759a59821cf" Sep 16 01:41:34.154235 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:34.154196 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55782ea1b697125425ec0e1ecf5b969c76aa29521da26c5850c9d759a59821cf"} err="failed to get container status \"55782ea1b697125425ec0e1ecf5b969c76aa29521da26c5850c9d759a59821cf\": rpc error: code = NotFound desc = could not find container \"55782ea1b697125425ec0e1ecf5b969c76aa29521da26c5850c9d759a59821cf\": container with ID starting with 55782ea1b697125425ec0e1ecf5b969c76aa29521da26c5850c9d759a59821cf not found: ID does not exist" Sep 16 01:41:34.154526 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:41:34.154503 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9138-5c3300\" is forbidden: User \"system:node:ip-10-0-137-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-55.ec2.internal' and this object" podUID="3c217985-0d86-4823-b953-41d9e4854290" pod="opendatahub/test-curl-9138-5c3300" Sep 16 01:41:34.294497 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:34.294467 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c217985-0d86-4823-b953-41d9e4854290" path="/var/lib/kubelet/pods/3c217985-0d86-4823-b953-41d9e4854290/volumes" Sep 16 01:41:36.634976 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:36.634945 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:38.555691 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:38.555660 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:39.189769 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:39.189715 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:41.128071 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:41.128033 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:41.175274 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:41.175246 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:41.524978 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:41.524897 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:41:44.790031 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:44.790001 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:47.840456 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:47.840421 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" podUID="1b596e9e-6243-4e33-ad5c-a066d92903f8" containerName="main" containerID="cri-o://88bf458bdc7507c69de011477fb24ecb556a55b6954b483a9c321735caacd483" gracePeriod=45 Sep 16 01:41:48.098430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.098404 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:41:48.231518 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.231428 2558 generic.go:356] "Generic (PLEG): container finished" podID="1b596e9e-6243-4e33-ad5c-a066d92903f8" containerID="88bf458bdc7507c69de011477fb24ecb556a55b6954b483a9c321735caacd483" exitCode=0 Sep 16 01:41:48.231518 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.231477 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" Sep 16 01:41:48.231694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.231518 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" event={"ID":"1b596e9e-6243-4e33-ad5c-a066d92903f8","Type":"ContainerDied","Data":"88bf458bdc7507c69de011477fb24ecb556a55b6954b483a9c321735caacd483"} Sep 16 01:41:48.231694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.231559 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc" event={"ID":"1b596e9e-6243-4e33-ad5c-a066d92903f8","Type":"ContainerDied","Data":"f0d9119a90c65eb64c44b01acfa87c6b4e4b27025087491723572570749684f2"} Sep 16 01:41:48.231694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.231581 2558 scope.go:117] "RemoveContainer" containerID="88bf458bdc7507c69de011477fb24ecb556a55b6954b483a9c321735caacd483" Sep 16 01:41:48.240538 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.240520 2558 scope.go:117] "RemoveContainer" containerID="88bf458bdc7507c69de011477fb24ecb556a55b6954b483a9c321735caacd483" Sep 16 01:41:48.240837 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:41:48.240820 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bf458bdc7507c69de011477fb24ecb556a55b6954b483a9c321735caacd483\": container with ID starting with 88bf458bdc7507c69de011477fb24ecb556a55b6954b483a9c321735caacd483 not found: ID does not exist" containerID="88bf458bdc7507c69de011477fb24ecb556a55b6954b483a9c321735caacd483" Sep 16 01:41:48.240906 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.240848 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bf458bdc7507c69de011477fb24ecb556a55b6954b483a9c321735caacd483"} err="failed to get container status \"88bf458bdc7507c69de011477fb24ecb556a55b6954b483a9c321735caacd483\": rpc error: code = NotFound desc = could not find container \"88bf458bdc7507c69de011477fb24ecb556a55b6954b483a9c321735caacd483\": container with ID starting with 88bf458bdc7507c69de011477fb24ecb556a55b6954b483a9c321735caacd483 not found: ID does not exist" Sep 16 01:41:48.253225 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.253209 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-tmp-dir\") pod \"1b596e9e-6243-4e33-ad5c-a066d92903f8\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " Sep 16 01:41:48.253288 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.253238 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b596e9e-6243-4e33-ad5c-a066d92903f8-tls-certs\") pod \"1b596e9e-6243-4e33-ad5c-a066d92903f8\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " Sep 16 01:41:48.253288 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.253268 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nc7c\" (UniqueName: \"kubernetes.io/projected/1b596e9e-6243-4e33-ad5c-a066d92903f8-kube-api-access-2nc7c\") pod \"1b596e9e-6243-4e33-ad5c-a066d92903f8\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " Sep 16 01:41:48.253352 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.253305 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-home\") pod \"1b596e9e-6243-4e33-ad5c-a066d92903f8\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " Sep 16 01:41:48.253393 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.253379 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-dshm\") pod \"1b596e9e-6243-4e33-ad5c-a066d92903f8\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " Sep 16 01:41:48.253442 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.253412 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-model-cache\") pod \"1b596e9e-6243-4e33-ad5c-a066d92903f8\" (UID: \"1b596e9e-6243-4e33-ad5c-a066d92903f8\") " Sep 16 01:41:48.253523 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.253504 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b596e9e-6243-4e33-ad5c-a066d92903f8" (UID: "1b596e9e-6243-4e33-ad5c-a066d92903f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:41:48.253675 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.253627 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-home" (OuterVolumeSpecName: "home") pod "1b596e9e-6243-4e33-ad5c-a066d92903f8" (UID: "1b596e9e-6243-4e33-ad5c-a066d92903f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:41:48.253751 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.253680 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b596e9e-6243-4e33-ad5c-a066d92903f8" (UID: "1b596e9e-6243-4e33-ad5c-a066d92903f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:41:48.253792 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.253777 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-tmp-dir\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:48.253792 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.253788 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-home\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:48.253859 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.253798 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-model-cache\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:48.255476 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.255447 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b596e9e-6243-4e33-ad5c-a066d92903f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b596e9e-6243-4e33-ad5c-a066d92903f8" (UID: "1b596e9e-6243-4e33-ad5c-a066d92903f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:41:48.255561 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.255493 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-dshm" (OuterVolumeSpecName: "dshm") pod "1b596e9e-6243-4e33-ad5c-a066d92903f8" (UID: "1b596e9e-6243-4e33-ad5c-a066d92903f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:41:48.255561 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.255539 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b596e9e-6243-4e33-ad5c-a066d92903f8-kube-api-access-2nc7c" (OuterVolumeSpecName: "kube-api-access-2nc7c") pod "1b596e9e-6243-4e33-ad5c-a066d92903f8" (UID: "1b596e9e-6243-4e33-ad5c-a066d92903f8"). InnerVolumeSpecName "kube-api-access-2nc7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:41:48.354441 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.354415 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b596e9e-6243-4e33-ad5c-a066d92903f8-tls-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:48.354441 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.354442 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nc7c\" (UniqueName: \"kubernetes.io/projected/1b596e9e-6243-4e33-ad5c-a066d92903f8-kube-api-access-2nc7c\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:48.354599 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.354450 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b596e9e-6243-4e33-ad5c-a066d92903f8-dshm\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:41:48.546698 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.546622 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc"] Sep 16 01:41:48.552214 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:48.552187 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/sub-test-model-c0ec73-kserve-7f5bdb645-2gcmc"] Sep 16 01:41:50.237455 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:50.237422 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:50.294860 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:50.294833 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b596e9e-6243-4e33-ad5c-a066d92903f8" path="/var/lib/kubelet/pods/1b596e9e-6243-4e33-ad5c-a066d92903f8/volumes" Sep 16 01:41:51.497611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:51.497577 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:51.804274 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:51.804180 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:51.902693 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:51.902664 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:52.003495 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:52.003468 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:52.670430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:52.670394 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:52.769403 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:52.769371 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:52.869981 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:52.869943 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:54.269637 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:54.269581 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:56.421575 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.421537 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8"] Sep 16 01:41:56.422180 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.422145 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c217985-0d86-4823-b953-41d9e4854290" containerName="test-curl-9138-5c3300" Sep 16 01:41:56.422242 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.422188 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c217985-0d86-4823-b953-41d9e4854290" containerName="test-curl-9138-5c3300" Sep 16 01:41:56.422242 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.422217 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b596e9e-6243-4e33-ad5c-a066d92903f8" containerName="main" Sep 16 01:41:56.422242 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.422226 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b596e9e-6243-4e33-ad5c-a066d92903f8" containerName="main" Sep 16 01:41:56.422417 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.422373 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c217985-0d86-4823-b953-41d9e4854290" containerName="test-curl-9138-5c3300" Sep 16 01:41:56.422417 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.422390 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b596e9e-6243-4e33-ad5c-a066d92903f8" containerName="main" Sep 16 01:41:56.426972 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.426951 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.429635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.429406 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c0ec73\"/\"default-dockercfg-p4jqz\"" Sep 16 01:41:56.429635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.429431 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c0ec73\"/\"openshift-service-ca.crt\"" Sep 16 01:41:56.429635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.429406 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c0ec73\"/\"rate-test-model-c0ec73-kserve-self-signed-certs\"" Sep 16 01:41:56.430057 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.430036 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c0ec73\"/\"kube-root-ca.crt\"" Sep 16 01:41:56.431992 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.431965 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8"] Sep 16 01:41:56.447825 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.447792 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:56.526711 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.526680 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-model-cache\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.526860 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.526802 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-home\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.526860 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.526838 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-tmp-dir\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.526961 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.526860 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5q5z\" (UniqueName: \"kubernetes.io/projected/b5847e68-1bab-4ce1-802a-d5056d769be0-kube-api-access-x5q5z\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.526961 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.526880 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-dshm\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.527043 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.526970 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5847e68-1bab-4ce1-802a-d5056d769be0-tls-certs\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.628571 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.628536 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-home\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.628571 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.628575 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-tmp-dir\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.628793 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.628593 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5q5z\" (UniqueName: \"kubernetes.io/projected/b5847e68-1bab-4ce1-802a-d5056d769be0-kube-api-access-x5q5z\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.628793 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.628610 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-dshm\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.628793 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.628657 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5847e68-1bab-4ce1-802a-d5056d769be0-tls-certs\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.628793 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.628731 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-model-cache\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.629000 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.628979 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-home\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.629045 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.629009 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-tmp-dir\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.629082 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.629059 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-model-cache\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.631039 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.631011 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-dshm\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.631340 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.631321 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5847e68-1bab-4ce1-802a-d5056d769be0-tls-certs\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.635639 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.635614 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5q5z\" (UniqueName: \"kubernetes.io/projected/b5847e68-1bab-4ce1-802a-d5056d769be0-kube-api-access-x5q5z\") pod \"rate-test-model-c0ec73-kserve-6788c47df5-8qfv8\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:56.739675 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:56.739598 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:41:57.011153 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:57.011126 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8"] Sep 16 01:41:57.013940 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:41:57.013905 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5847e68_1bab_4ce1_802a_d5056d769be0.slice/crio-8cd71285ff8495c0c9d76731681cc0ee21c0db0073521333f3ec69f5d36a3aa6 WatchSource:0}: Error finding container 8cd71285ff8495c0c9d76731681cc0ee21c0db0073521333f3ec69f5d36a3aa6: Status 404 returned error can't find the container with id 8cd71285ff8495c0c9d76731681cc0ee21c0db0073521333f3ec69f5d36a3aa6 Sep 16 01:41:57.292541 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:57.292456 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" event={"ID":"b5847e68-1bab-4ce1-802a-d5056d769be0","Type":"ContainerStarted","Data":"0c8e133c553e58aea859f00e25f72ea70f6cc9e0ff51bfcef2991c8412bcabcb"} Sep 16 01:41:57.292541 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:57.292501 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" event={"ID":"b5847e68-1bab-4ce1-802a-d5056d769be0","Type":"ContainerStarted","Data":"8cd71285ff8495c0c9d76731681cc0ee21c0db0073521333f3ec69f5d36a3aa6"} Sep 16 01:41:57.307660 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:57.307585 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" podStartSLOduration=1.307566741 podStartE2EDuration="1.307566741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:41:57.306392364 +0000 UTC m=+1547.558814952" watchObservedRunningTime="2026-09-16 01:41:57.307566741 +0000 UTC m=+1547.559989331" Sep 16 01:41:58.011816 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:58.011784 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:41:59.870023 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:59.869993 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:41:59.970088 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:41:59.970062 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:00.562009 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:00.561972 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:06.133060 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:06.133032 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:06.240490 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:06.240454 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:06.333295 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:06.333262 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:06.739818 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:06.739787 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:42:06.740003 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:06.739851 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:42:06.752423 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:06.752397 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:42:07.359554 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:07.359523 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:42:12.513676 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:12.513641 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:42:16.860959 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:16.860917 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:17.106662 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:17.106635 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:23.415271 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:23.415239 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:42:34.709021 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:34.708989 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:42:36.029254 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:36.029221 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:38.073091 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:38.073054 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:40.911678 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:40.911645 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:41.798530 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:41.798499 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8"] Sep 16 01:42:41.810495 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:41.810471 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:42.270665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:42.270632 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:42.572587 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:42.572500 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:42.745206 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:42.745134 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:43.971537 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:43.971505 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:44.071735 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.071709 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:44.346052 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.345978 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:44.563260 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.563208 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:44.629570 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.629533 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42"] Sep 16 01:42:44.633400 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.633378 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.635808 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.635787 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7d0959-openshift-default-dockercfg-gkkhv\"" Sep 16 01:42:44.643783 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.643758 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42"] Sep 16 01:42:44.662023 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.661996 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:44.755511 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.755476 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75cba283-9c47-467b-9ea1-c82250b6e394-istio-token\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.755511 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.755512 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-istio-envoy\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.755779 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.755645 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-istio-data\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.755779 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.755723 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cba283-9c47-467b-9ea1-c82250b6e394-istiod-ca-cert\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.755779 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.755756 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-workload-socket\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.755931 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.755787 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-credential-socket\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.755931 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.755826 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4xqj\" (UniqueName: \"kubernetes.io/projected/75cba283-9c47-467b-9ea1-c82250b6e394-kube-api-access-z4xqj\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.755931 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.755902 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cba283-9c47-467b-9ea1-c82250b6e394-istio-podinfo\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.756082 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.755968 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-workload-certs\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.856742 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.856705 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-workload-certs\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.856923 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.856792 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75cba283-9c47-467b-9ea1-c82250b6e394-istio-token\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.856923 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.856824 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-istio-envoy\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.857111 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.856974 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-istio-data\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.857111 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.857034 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cba283-9c47-467b-9ea1-c82250b6e394-istiod-ca-cert\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.857111 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.857074 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-workload-socket\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.857111 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.857097 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-workload-certs\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.857383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.857126 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-credential-socket\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.857383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.857199 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4xqj\" (UniqueName: \"kubernetes.io/projected/75cba283-9c47-467b-9ea1-c82250b6e394-kube-api-access-z4xqj\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.857383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.857273 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cba283-9c47-467b-9ea1-c82250b6e394-istio-podinfo\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.857383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.857370 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-istio-data\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.857758 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.857485 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-credential-socket\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.857758 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.857689 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cba283-9c47-467b-9ea1-c82250b6e394-istiod-ca-cert\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.857878 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.857814 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-workload-socket\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.859617 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.859581 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-istio-envoy\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.860071 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.860047 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cba283-9c47-467b-9ea1-c82250b6e394-istio-podinfo\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.865303 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.865258 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4xqj\" (UniqueName: \"kubernetes.io/projected/75cba283-9c47-467b-9ea1-c82250b6e394-kube-api-access-z4xqj\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.865823 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.865799 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75cba283-9c47-467b-9ea1-c82250b6e394-istio-token\") pod \"e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:44.897565 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.897487 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:44.945526 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:44.945502 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:45.231768 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:45.231737 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42"] Sep 16 01:42:45.233139 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:42:45.233110 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cba283_9c47_467b_9ea1_c82250b6e394.slice/crio-b2269b9f60209a25094a1ba250a9b785cb24baf87fe8b761c219b4ece5e88921 WatchSource:0}: Error finding container b2269b9f60209a25094a1ba250a9b785cb24baf87fe8b761c219b4ece5e88921: Status 404 returned error can't find the container with id b2269b9f60209a25094a1ba250a9b785cb24baf87fe8b761c219b4ece5e88921 Sep 16 01:42:45.235350 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:45.235315 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:42:45.235449 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:45.235392 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:42:45.235449 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:45.235434 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:42:45.254116 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:45.254091 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:45.322055 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:45.322018 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:45.570557 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:45.570472 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" event={"ID":"75cba283-9c47-467b-9ea1-c82250b6e394","Type":"ContainerStarted","Data":"548118c5c9c236a6dca7621f13826cd64e61f41cef097029726e21f784d29c42"} Sep 16 01:42:45.570557 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:45.570525 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" event={"ID":"75cba283-9c47-467b-9ea1-c82250b6e394","Type":"ContainerStarted","Data":"b2269b9f60209a25094a1ba250a9b785cb24baf87fe8b761c219b4ece5e88921"} Sep 16 01:42:45.593099 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:45.593043 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" podStartSLOduration=1.593027795 podStartE2EDuration="1.593027795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:42:45.590432168 +0000 UTC m=+1595.842854777" watchObservedRunningTime="2026-09-16 01:42:45.593027795 +0000 UTC m=+1595.845450384" Sep 16 01:42:45.945801 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:45.945768 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:46.150552 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:46.150521 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:46.952048 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:46.952019 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:47.161135 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:47.161103 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:47.539829 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:47.539792 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:47.583699 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:47.583672 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:47.584714 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:47.584692 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:42:47.612593 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:47.612565 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:47.725635 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:47.725601 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:42:48.072932 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:48.072903 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:48.128482 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:48.128454 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:48.275695 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:48.275649 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:48.725759 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:48.725716 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:48.825967 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:48.825934 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:48.836374 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:48.836338 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89"] Sep 16 01:42:48.840117 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:48.840099 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" Sep 16 01:42:48.842319 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:48.842302 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8d230b-dockercfg-9mq7j\"" Sep 16 01:42:48.842407 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:48.842347 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8d230b\"" Sep 16 01:42:48.848029 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:48.848008 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89"] Sep 16 01:42:49.000870 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:49.000779 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fd6a87-8388-4175-b327-15dee2429085-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89\" (UID: \"32fd6a87-8388-4175-b327-15dee2429085\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" Sep 16 01:42:49.000870 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:49.000813 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpzwq\" (UniqueName: \"kubernetes.io/projected/32fd6a87-8388-4175-b327-15dee2429085-kube-api-access-gpzwq\") pod \"payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89\" (UID: \"32fd6a87-8388-4175-b327-15dee2429085\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" Sep 16 01:42:49.082616 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:49.082580 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:49.101871 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:49.101845 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fd6a87-8388-4175-b327-15dee2429085-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89\" (UID: \"32fd6a87-8388-4175-b327-15dee2429085\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" Sep 16 01:42:49.102019 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:49.101879 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpzwq\" (UniqueName: \"kubernetes.io/projected/32fd6a87-8388-4175-b327-15dee2429085-kube-api-access-gpzwq\") pod \"payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89\" (UID: \"32fd6a87-8388-4175-b327-15dee2429085\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" Sep 16 01:42:49.102616 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:49.102597 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fd6a87-8388-4175-b327-15dee2429085-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89\" (UID: \"32fd6a87-8388-4175-b327-15dee2429085\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" Sep 16 01:42:49.110105 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:49.110073 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpzwq\" (UniqueName: \"kubernetes.io/projected/32fd6a87-8388-4175-b327-15dee2429085-kube-api-access-gpzwq\") pod \"payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89\" (UID: \"32fd6a87-8388-4175-b327-15dee2429085\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" Sep 16 01:42:49.153052 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:49.153020 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" Sep 16 01:42:49.428096 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:49.428067 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89"] Sep 16 01:42:49.428825 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:42:49.428800 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fd6a87_8388_4175_b327_15dee2429085.slice/crio-0b1ad2d5f171140d4d6f04fe15f9f415941a69287a5ad4af79192988ea993e75 WatchSource:0}: Error finding container 0b1ad2d5f171140d4d6f04fe15f9f415941a69287a5ad4af79192988ea993e75: Status 404 returned error can't find the container with id 0b1ad2d5f171140d4d6f04fe15f9f415941a69287a5ad4af79192988ea993e75 Sep 16 01:42:49.605220 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:49.605170 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" event={"ID":"32fd6a87-8388-4175-b327-15dee2429085","Type":"ContainerStarted","Data":"799070baefe86528eff4b45a3641e3429f424310d97ac1865928406259137ec0"} Sep 16 01:42:49.605434 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:49.605260 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" event={"ID":"32fd6a87-8388-4175-b327-15dee2429085","Type":"ContainerStarted","Data":"0b1ad2d5f171140d4d6f04fe15f9f415941a69287a5ad4af79192988ea993e75"} Sep 16 01:42:49.621205 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:49.621139 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" podStartSLOduration=1.62112583 podStartE2EDuration="1.62112583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:42:49.619501393 +0000 UTC m=+1599.871924002" watchObservedRunningTime="2026-09-16 01:42:49.62112583 +0000 UTC m=+1599.873548418" Sep 16 01:42:49.782908 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:49.782768 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:50.611123 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:50.611092 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" Sep 16 01:42:50.688738 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:50.688704 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:50.787115 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:50.787081 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:51.124826 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.124787 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:51.230655 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.230617 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:51.235660 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.235635 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t"] Sep 16 01:42:51.239268 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.239251 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" Sep 16 01:42:51.241374 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.241353 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7d0959\"" Sep 16 01:42:51.241484 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.241356 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7d0959-dockercfg-fwgzk\"" Sep 16 01:42:51.248697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.248671 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t"] Sep 16 01:42:51.424726 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.424643 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5569bd81-5f3b-462d-860c-ebdc3d7814ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t\" (UID: \"5569bd81-5f3b-462d-860c-ebdc3d7814ff\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" Sep 16 01:42:51.424726 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.424684 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzj52\" (UniqueName: \"kubernetes.io/projected/5569bd81-5f3b-462d-860c-ebdc3d7814ff-kube-api-access-fzj52\") pod \"payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t\" (UID: \"5569bd81-5f3b-462d-860c-ebdc3d7814ff\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" Sep 16 01:42:51.525457 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.525417 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5569bd81-5f3b-462d-860c-ebdc3d7814ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t\" (UID: \"5569bd81-5f3b-462d-860c-ebdc3d7814ff\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" Sep 16 01:42:51.525457 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.525458 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzj52\" (UniqueName: \"kubernetes.io/projected/5569bd81-5f3b-462d-860c-ebdc3d7814ff-kube-api-access-fzj52\") pod \"payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t\" (UID: \"5569bd81-5f3b-462d-860c-ebdc3d7814ff\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" Sep 16 01:42:51.526222 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.526199 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5569bd81-5f3b-462d-860c-ebdc3d7814ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t\" (UID: \"5569bd81-5f3b-462d-860c-ebdc3d7814ff\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" Sep 16 01:42:51.532551 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.532528 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzj52\" (UniqueName: \"kubernetes.io/projected/5569bd81-5f3b-462d-860c-ebdc3d7814ff-kube-api-access-fzj52\") pod \"payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t\" (UID: \"5569bd81-5f3b-462d-860c-ebdc3d7814ff\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" Sep 16 01:42:51.550270 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.550241 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" Sep 16 01:42:51.579615 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.579572 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:51.680518 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.680441 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:51.826957 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:51.826934 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t"] Sep 16 01:42:51.828654 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:42:51.828622 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5569bd81_5f3b_462d_860c_ebdc3d7814ff.slice/crio-21f30ffd21c354292a7175f4d1088c89447d95614a0cd8fe0c6098053ec6dfb3 WatchSource:0}: Error finding container 21f30ffd21c354292a7175f4d1088c89447d95614a0cd8fe0c6098053ec6dfb3: Status 404 returned error can't find the container with id 21f30ffd21c354292a7175f4d1088c89447d95614a0cd8fe0c6098053ec6dfb3 Sep 16 01:42:52.363774 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:52.363737 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:52.627763 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:52.627677 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" event={"ID":"5569bd81-5f3b-462d-860c-ebdc3d7814ff","Type":"ContainerStarted","Data":"ac24b2ef96c63dc152fc9af7d538ee4518a6ea84db5dd44628c9281eea497d7b"} Sep 16 01:42:52.627763 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:52.627728 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" event={"ID":"5569bd81-5f3b-462d-860c-ebdc3d7814ff","Type":"ContainerStarted","Data":"21f30ffd21c354292a7175f4d1088c89447d95614a0cd8fe0c6098053ec6dfb3"} Sep 16 01:42:52.627763 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:52.627758 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" Sep 16 01:42:52.643053 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:52.642996 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" podStartSLOduration=1.642976461 podStartE2EDuration="1.642976461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:42:52.642704858 +0000 UTC m=+1602.895127449" watchObservedRunningTime="2026-09-16 01:42:52.642976461 +0000 UTC m=+1602.895399073" Sep 16 01:42:52.984097 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:52.984006 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:53.609266 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:53.609236 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:42:56.818534 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:56.818487 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" podUID="b5847e68-1bab-4ce1-802a-d5056d769be0" containerName="main" containerID="cri-o://0c8e133c553e58aea859f00e25f72ea70f6cc9e0ff51bfcef2991c8412bcabcb" gracePeriod=45 Sep 16 01:42:57.076911 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.076855 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:42:57.173878 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.173848 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-home\") pod \"b5847e68-1bab-4ce1-802a-d5056d769be0\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " Sep 16 01:42:57.174045 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.173884 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-tmp-dir\") pod \"b5847e68-1bab-4ce1-802a-d5056d769be0\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " Sep 16 01:42:57.174045 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.173941 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5q5z\" (UniqueName: \"kubernetes.io/projected/b5847e68-1bab-4ce1-802a-d5056d769be0-kube-api-access-x5q5z\") pod \"b5847e68-1bab-4ce1-802a-d5056d769be0\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " Sep 16 01:42:57.174045 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.173967 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5847e68-1bab-4ce1-802a-d5056d769be0-tls-certs\") pod \"b5847e68-1bab-4ce1-802a-d5056d769be0\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " Sep 16 01:42:57.174045 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.174013 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-model-cache\") pod \"b5847e68-1bab-4ce1-802a-d5056d769be0\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " Sep 16 01:42:57.174300 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.174133 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-home" (OuterVolumeSpecName: "home") pod "b5847e68-1bab-4ce1-802a-d5056d769be0" (UID: "b5847e68-1bab-4ce1-802a-d5056d769be0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:42:57.174300 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.174142 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-dshm\") pod \"b5847e68-1bab-4ce1-802a-d5056d769be0\" (UID: \"b5847e68-1bab-4ce1-802a-d5056d769be0\") " Sep 16 01:42:57.174393 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.174279 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5847e68-1bab-4ce1-802a-d5056d769be0" (UID: "b5847e68-1bab-4ce1-802a-d5056d769be0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:42:57.174393 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.174365 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5847e68-1bab-4ce1-802a-d5056d769be0" (UID: "b5847e68-1bab-4ce1-802a-d5056d769be0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:42:57.174519 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.174501 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-home\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:42:57.174578 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.174525 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-tmp-dir\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:42:57.174578 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.174539 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-model-cache\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:42:57.176299 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.176271 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5847e68-1bab-4ce1-802a-d5056d769be0-kube-api-access-x5q5z" (OuterVolumeSpecName: "kube-api-access-x5q5z") pod "b5847e68-1bab-4ce1-802a-d5056d769be0" (UID: "b5847e68-1bab-4ce1-802a-d5056d769be0"). InnerVolumeSpecName "kube-api-access-x5q5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:42:57.176299 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.176286 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5847e68-1bab-4ce1-802a-d5056d769be0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5847e68-1bab-4ce1-802a-d5056d769be0" (UID: "b5847e68-1bab-4ce1-802a-d5056d769be0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:42:57.176299 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.176279 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-dshm" (OuterVolumeSpecName: "dshm") pod "b5847e68-1bab-4ce1-802a-d5056d769be0" (UID: "b5847e68-1bab-4ce1-802a-d5056d769be0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:42:57.275498 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.275476 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5847e68-1bab-4ce1-802a-d5056d769be0-dshm\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:42:57.275498 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.275499 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5q5z\" (UniqueName: \"kubernetes.io/projected/b5847e68-1bab-4ce1-802a-d5056d769be0-kube-api-access-x5q5z\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:42:57.275665 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.275508 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5847e68-1bab-4ce1-802a-d5056d769be0-tls-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:42:57.658797 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.658756 2558 generic.go:356] "Generic (PLEG): container finished" podID="b5847e68-1bab-4ce1-802a-d5056d769be0" containerID="0c8e133c553e58aea859f00e25f72ea70f6cc9e0ff51bfcef2991c8412bcabcb" exitCode=0 Sep 16 01:42:57.658797 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.658799 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" Sep 16 01:42:57.659040 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.658843 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" event={"ID":"b5847e68-1bab-4ce1-802a-d5056d769be0","Type":"ContainerDied","Data":"0c8e133c553e58aea859f00e25f72ea70f6cc9e0ff51bfcef2991c8412bcabcb"} Sep 16 01:42:57.659040 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.658888 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8" event={"ID":"b5847e68-1bab-4ce1-802a-d5056d769be0","Type":"ContainerDied","Data":"8cd71285ff8495c0c9d76731681cc0ee21c0db0073521333f3ec69f5d36a3aa6"} Sep 16 01:42:57.659040 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.658908 2558 scope.go:117] "RemoveContainer" containerID="0c8e133c553e58aea859f00e25f72ea70f6cc9e0ff51bfcef2991c8412bcabcb" Sep 16 01:42:57.670591 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.670575 2558 scope.go:117] "RemoveContainer" containerID="0c8e133c553e58aea859f00e25f72ea70f6cc9e0ff51bfcef2991c8412bcabcb" Sep 16 01:42:57.670855 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:42:57.670835 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8e133c553e58aea859f00e25f72ea70f6cc9e0ff51bfcef2991c8412bcabcb\": container with ID starting with 0c8e133c553e58aea859f00e25f72ea70f6cc9e0ff51bfcef2991c8412bcabcb not found: ID does not exist" containerID="0c8e133c553e58aea859f00e25f72ea70f6cc9e0ff51bfcef2991c8412bcabcb" Sep 16 01:42:57.670926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.670866 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8e133c553e58aea859f00e25f72ea70f6cc9e0ff51bfcef2991c8412bcabcb"} err="failed to get container status \"0c8e133c553e58aea859f00e25f72ea70f6cc9e0ff51bfcef2991c8412bcabcb\": rpc error: code = NotFound desc = could not find container \"0c8e133c553e58aea859f00e25f72ea70f6cc9e0ff51bfcef2991c8412bcabcb\": container with ID starting with 0c8e133c553e58aea859f00e25f72ea70f6cc9e0ff51bfcef2991c8412bcabcb not found: ID does not exist" Sep 16 01:42:57.680319 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.680300 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8"] Sep 16 01:42:57.685988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:57.685967 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c0ec73/rate-test-model-c0ec73-kserve-6788c47df5-8qfv8"] Sep 16 01:42:58.296993 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:42:58.296958 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5847e68-1bab-4ce1-802a-d5056d769be0" path="/var/lib/kubelet/pods/b5847e68-1bab-4ce1-802a-d5056d769be0/volumes" Sep 16 01:43:01.622122 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:01.622039 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" Sep 16 01:43:03.635854 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:03.635816 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" Sep 16 01:43:07.737348 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:07.737317 2558 generic.go:356] "Generic (PLEG): container finished" podID="75cba283-9c47-467b-9ea1-c82250b6e394" containerID="548118c5c9c236a6dca7621f13826cd64e61f41cef097029726e21f784d29c42" exitCode=0 Sep 16 01:43:07.737767 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:07.737361 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" event={"ID":"75cba283-9c47-467b-9ea1-c82250b6e394","Type":"ContainerDied","Data":"548118c5c9c236a6dca7621f13826cd64e61f41cef097029726e21f784d29c42"} Sep 16 01:43:07.737767 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:07.737698 2558 scope.go:117] "RemoveContainer" containerID="548118c5c9c236a6dca7621f13826cd64e61f41cef097029726e21f784d29c42" Sep 16 01:43:07.738887 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:07.738853 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:07.738988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:07.738923 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:07.738988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:07.738954 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:08.015936 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:08.015857 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:43:08.745240 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:08.745208 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" event={"ID":"75cba283-9c47-467b-9ea1-c82250b6e394","Type":"ContainerStarted","Data":"91e657332d381836ae295b03d1b50653851f5753459f0bf16e1636d9324c4961"} Sep 16 01:43:08.946501 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:08.946466 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:43:08.946722 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:08.946698 2558 patch_prober.go:28] interesting pod/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 16 01:43:08.946776 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:08.946750 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" podUID="75cba283-9c47-467b-9ea1-c82250b6e394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 16 01:43:09.086218 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:09.086103 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:09.946104 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:09.946059 2558 patch_prober.go:28] interesting pod/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 16 01:43:09.946509 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:09.946125 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" podUID="75cba283-9c47-467b-9ea1-c82250b6e394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 16 01:43:10.286682 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:10.286584 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:10.385566 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:10.385524 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:10.946821 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:10.946784 2558 patch_prober.go:28] interesting pod/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 16 01:43:10.947217 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:10.946845 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" podUID="75cba283-9c47-467b-9ea1-c82250b6e394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 16 01:43:10.986644 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:10.986611 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:11.086812 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.086770 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:11.279710 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.279604 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8"] Sep 16 01:43:11.280469 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.280448 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5847e68-1bab-4ce1-802a-d5056d769be0" containerName="main" Sep 16 01:43:11.280647 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.280632 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5847e68-1bab-4ce1-802a-d5056d769be0" containerName="main" Sep 16 01:43:11.280988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.280959 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5847e68-1bab-4ce1-802a-d5056d769be0" containerName="main" Sep 16 01:43:11.286200 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.286177 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.290706 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.290686 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a42101-openshift-default-dockercfg-blxs6\"" Sep 16 01:43:11.297088 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.297058 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8"] Sep 16 01:43:11.310286 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.310263 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-credential-socket\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.310433 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.310418 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-data\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.310573 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.310563 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istiod-ca-cert\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.310669 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.310654 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8b29\" (UniqueName: \"kubernetes.io/projected/8c44983e-f5c6-4c73-b8a1-844e47b3390e-kube-api-access-p8b29\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.310785 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.310771 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-workload-certs\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.310862 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.310838 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-podinfo\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.310918 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.310880 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-envoy\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.311015 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.310996 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-workload-socket\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.311074 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.311028 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-token\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.318852 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.318810 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:11.413027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.412240 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istiod-ca-cert\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.413027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.412313 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8b29\" (UniqueName: \"kubernetes.io/projected/8c44983e-f5c6-4c73-b8a1-844e47b3390e-kube-api-access-p8b29\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.413027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.412382 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-workload-certs\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.413027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.412444 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-podinfo\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.413027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.412482 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-envoy\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.413027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.412563 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-workload-socket\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.413027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.412602 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-token\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.413027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.412668 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-credential-socket\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.413027 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.412694 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-data\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.413686 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.413534 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istiod-ca-cert\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.413686 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.413595 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-data\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.414183 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.413946 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-credential-socket\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.414183 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.414071 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-workload-socket\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.414183 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.414080 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-workload-certs\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.415954 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.415920 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-envoy\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.416309 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.416290 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-podinfo\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.420623 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.420599 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-token\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.420918 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.420900 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8b29\" (UniqueName: \"kubernetes.io/projected/8c44983e-f5c6-4c73-b8a1-844e47b3390e-kube-api-access-p8b29\") pod \"e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.543150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.543040 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:11.606705 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.606633 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:11.790185 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.789642 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:11.898394 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.898367 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8"] Sep 16 01:43:11.900814 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:43:11.900785 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c44983e_f5c6_4c73_b8a1_844e47b3390e.slice/crio-ce58f231d5bb634141397390721a0082c59c3cc5550e257044bc2ff85e19c683 WatchSource:0}: Error finding container ce58f231d5bb634141397390721a0082c59c3cc5550e257044bc2ff85e19c683: Status 404 returned error can't find the container with id ce58f231d5bb634141397390721a0082c59c3cc5550e257044bc2ff85e19c683 Sep 16 01:43:11.903722 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.903682 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:11.903837 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.903792 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:11.903902 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.903835 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:11.946142 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.946113 2558 patch_prober.go:28] interesting pod/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 16 01:43:11.946254 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:11.946180 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" podUID="75cba283-9c47-467b-9ea1-c82250b6e394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 16 01:43:12.794929 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:12.794895 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" event={"ID":"8c44983e-f5c6-4c73-b8a1-844e47b3390e","Type":"ContainerStarted","Data":"9eb6bd558cc77bcab2a0364781a3a66f85ed1f2bbb6fa9cb067766657539f460"} Sep 16 01:43:12.795507 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:12.795483 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" event={"ID":"8c44983e-f5c6-4c73-b8a1-844e47b3390e","Type":"ContainerStarted","Data":"ce58f231d5bb634141397390721a0082c59c3cc5550e257044bc2ff85e19c683"} Sep 16 01:43:12.815307 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:12.815249 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" podStartSLOduration=1.815230652 podStartE2EDuration="1.815230652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:43:12.81254303 +0000 UTC m=+1623.064965618" watchObservedRunningTime="2026-09-16 01:43:12.815230652 +0000 UTC m=+1623.067653243" Sep 16 01:43:12.877139 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:12.877092 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:12.946699 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:12.946668 2558 patch_prober.go:28] interesting pod/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 16 01:43:12.946890 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:12.946847 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" podUID="75cba283-9c47-467b-9ea1-c82250b6e394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 16 01:43:13.486981 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:13.486941 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:13.607233 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:13.607198 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:13.612656 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:13.612626 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:13.801313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:13.801212 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:13.802439 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:13.802412 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:14.029800 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:14.029767 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:43:14.030284 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:14.030265 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:43:14.031990 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:14.031966 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:43:14.360318 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:14.360273 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:14.877008 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:14.876979 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:14.974438 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:14.974382 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:15.152910 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:15.152792 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:15.958598 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:15.958570 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:16.725588 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:16.725559 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:17.564526 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:17.564498 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:17.665272 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:17.665239 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:18.137975 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.137941 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:18.235038 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.235009 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:18.339297 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.339258 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:18.419042 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.418970 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb"] Sep 16 01:43:18.422975 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.422955 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.427483 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.427458 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4e1413-openshift-default-dockercfg-hlfl7\"" Sep 16 01:43:18.437550 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.436016 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb"] Sep 16 01:43:18.441018 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.440995 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:18.484250 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.484223 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-token\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.484377 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.484260 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6k6f\" (UniqueName: \"kubernetes.io/projected/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-kube-api-access-k6k6f\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.484377 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.484280 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-workload-socket\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.484377 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.484296 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-data\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.484510 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.484394 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-credential-socket\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.484510 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.484426 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-workload-certs\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.484510 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.484455 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-podinfo\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.484510 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.484491 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istiod-ca-cert\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.484684 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.484525 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-envoy\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.585112 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.585079 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-podinfo\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.585112 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.585116 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istiod-ca-cert\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.585646 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.585141 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-envoy\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.585646 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.585264 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-token\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.585646 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.585290 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6k6f\" (UniqueName: \"kubernetes.io/projected/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-kube-api-access-k6k6f\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.585646 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.585307 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-workload-socket\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.585646 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.585322 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-data\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.585646 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.585369 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-credential-socket\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.585646 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.585396 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-workload-certs\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.586043 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.585770 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-workload-socket\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.586043 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.585855 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-credential-socket\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.586043 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.585908 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-data\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.586266 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.586129 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istiod-ca-cert\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.586327 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.586214 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-workload-certs\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.587937 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.587917 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-envoy\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.588083 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.588064 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-podinfo\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.593555 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.593530 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6k6f\" (UniqueName: \"kubernetes.io/projected/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-kube-api-access-k6k6f\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.593658 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.593614 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-token\") pod \"e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.631428 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.631401 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:18.733620 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.733290 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:18.742727 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.742702 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:18.803641 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.803587 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:18.859408 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.859375 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5"] Sep 16 01:43:18.863384 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.863342 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" Sep 16 01:43:18.865703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.865682 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a42101\"" Sep 16 01:43:18.865801 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.865687 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a42101-dockercfg-6pvjk\"" Sep 16 01:43:18.873558 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.873535 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5"] Sep 16 01:43:18.989089 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.989009 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5a3df4-818f-4c84-8187-c29a4ca5b7e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5\" (UID: \"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" Sep 16 01:43:18.989260 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:18.989120 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmkpz\" (UniqueName: \"kubernetes.io/projected/ab5a3df4-818f-4c84-8187-c29a4ca5b7e3-kube-api-access-gmkpz\") pod \"payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5\" (UID: \"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" Sep 16 01:43:19.060744 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.060716 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb"] Sep 16 01:43:19.061625 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:43:19.061598 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ab75d0_b5a4_44c5_b74d_8246e7ca5c4a.slice/crio-05f51fccfdf30f1015f2d1459306725229d12c72dd44059f9fc48fae5b547438 WatchSource:0}: Error finding container 05f51fccfdf30f1015f2d1459306725229d12c72dd44059f9fc48fae5b547438: Status 404 returned error can't find the container with id 05f51fccfdf30f1015f2d1459306725229d12c72dd44059f9fc48fae5b547438 Sep 16 01:43:19.064624 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.064581 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:19.064730 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.064669 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:19.064730 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.064711 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:19.089702 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.089675 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5a3df4-818f-4c84-8187-c29a4ca5b7e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5\" (UID: \"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" Sep 16 01:43:19.089821 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.089764 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmkpz\" (UniqueName: \"kubernetes.io/projected/ab5a3df4-818f-4c84-8187-c29a4ca5b7e3-kube-api-access-gmkpz\") pod \"payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5\" (UID: \"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" Sep 16 01:43:19.090413 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.090392 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5a3df4-818f-4c84-8187-c29a4ca5b7e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5\" (UID: \"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" Sep 16 01:43:19.106563 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.106545 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmkpz\" (UniqueName: \"kubernetes.io/projected/ab5a3df4-818f-4c84-8187-c29a4ca5b7e3-kube-api-access-gmkpz\") pod \"payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5\" (UID: \"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" Sep 16 01:43:19.163265 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.163238 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:19.176094 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.176070 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" Sep 16 01:43:19.456672 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.456648 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5"] Sep 16 01:43:19.458110 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:43:19.458072 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5a3df4_818f_4c84_8187_c29a4ca5b7e3.slice/crio-571829d2abc44d8be8888c337c2fb25e6c0863f4adf676f0eed8dd0d07c390c8 WatchSource:0}: Error finding container 571829d2abc44d8be8888c337c2fb25e6c0863f4adf676f0eed8dd0d07c390c8: Status 404 returned error can't find the container with id 571829d2abc44d8be8888c337c2fb25e6c0863f4adf676f0eed8dd0d07c390c8 Sep 16 01:43:19.842637 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.842538 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" event={"ID":"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a","Type":"ContainerStarted","Data":"461df808e25232900f5c87e59c8d5f53b99b1161a745f2fbd8a44ce67913f419"} Sep 16 01:43:19.842637 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.842608 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" event={"ID":"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a","Type":"ContainerStarted","Data":"05f51fccfdf30f1015f2d1459306725229d12c72dd44059f9fc48fae5b547438"} Sep 16 01:43:19.844407 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.844380 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" event={"ID":"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3","Type":"ContainerStarted","Data":"19f8cce45e7f1e4df908ebb7de7614b11e0443701c01c4563e7525a4db8eb161"} Sep 16 01:43:19.844525 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.844423 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" event={"ID":"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3","Type":"ContainerStarted","Data":"571829d2abc44d8be8888c337c2fb25e6c0863f4adf676f0eed8dd0d07c390c8"} Sep 16 01:43:19.844525 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.844506 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" Sep 16 01:43:19.865174 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.865101 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" podStartSLOduration=1.8650525770000002 podStartE2EDuration="1.865052577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:43:19.859242494 +0000 UTC m=+1630.111665084" watchObservedRunningTime="2026-09-16 01:43:19.865052577 +0000 UTC m=+1630.117475168" Sep 16 01:43:19.879747 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.879699 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" podStartSLOduration=1.8796852149999999 podStartE2EDuration="1.879685215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:43:19.877767183 +0000 UTC m=+1630.130189773" watchObservedRunningTime="2026-09-16 01:43:19.879685215 +0000 UTC m=+1630.132107803" Sep 16 01:43:19.964634 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:19.964598 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:20.743404 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:20.743371 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:20.748546 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:20.748521 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:20.850551 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:20.850518 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:20.851361 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:20.851344 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:21.143836 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:21.143777 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:21.959340 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:21.959304 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:22.058362 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:22.058327 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:22.422856 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:22.422821 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:43:22.765736 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:22.765641 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:23.758850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:23.758806 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:24.440718 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:24.440676 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:25.049422 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:25.049387 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:25.722306 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:25.722274 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:26.562089 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:26.561980 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:27.375087 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:27.375023 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:27.477939 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:27.477894 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:27.906911 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:27.906867 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:28.009689 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:28.009659 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:28.261991 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:28.261894 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:28.878415 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:28.878377 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:29.547659 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:29.547619 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:29.645373 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:29.645340 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:30.374261 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:30.374228 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:30.852816 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:30.852723 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" Sep 16 01:43:31.084343 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:31.084288 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:31.689972 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:31.689943 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:32.456858 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:32.456820 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:32.934185 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:32.934142 2558 generic.go:356] "Generic (PLEG): container finished" podID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerID="9eb6bd558cc77bcab2a0364781a3a66f85ed1f2bbb6fa9cb067766657539f460" exitCode=0 Sep 16 01:43:32.934348 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:32.934194 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" event={"ID":"8c44983e-f5c6-4c73-b8a1-844e47b3390e","Type":"ContainerDied","Data":"9eb6bd558cc77bcab2a0364781a3a66f85ed1f2bbb6fa9cb067766657539f460"} Sep 16 01:43:32.934575 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:32.934561 2558 scope.go:117] "RemoveContainer" containerID="9eb6bd558cc77bcab2a0364781a3a66f85ed1f2bbb6fa9cb067766657539f460" Sep 16 01:43:32.935715 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:32.935684 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:32.935836 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:32.935773 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:32.935836 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:32.935824 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:33.092680 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:33.092651 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:33.942008 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:33.941973 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" event={"ID":"8c44983e-f5c6-4c73-b8a1-844e47b3390e","Type":"ContainerStarted","Data":"708aa54ce25df0739d9d504cc018abfee76e09c010db1c0cef8078309fbac8e0"} Sep 16 01:43:34.606867 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:34.606825 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:34.607054 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:34.607034 2558 patch_prober.go:28] interesting pod/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 16 01:43:34.607111 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:34.607080 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 16 01:43:35.607006 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:35.606968 2558 patch_prober.go:28] interesting pod/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 16 01:43:35.607418 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:35.607031 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 16 01:43:36.607415 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:36.607377 2558 patch_prober.go:28] interesting pod/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 16 01:43:36.608028 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:36.607447 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 16 01:43:37.607405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:37.607361 2558 patch_prober.go:28] interesting pod/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 16 01:43:37.607620 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:37.607446 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 16 01:43:38.607068 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:38.607023 2558 patch_prober.go:28] interesting pod/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 16 01:43:38.607301 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:38.607092 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 16 01:43:39.070572 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:39.070502 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:39.611719 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:39.611688 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:39.612023 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:39.612002 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:39.612620 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:39.612604 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:43:40.305104 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:40.305075 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5_ab5a3df4-818f-4c84-8187-c29a4ca5b7e3/payload-processing/0.log" Sep 16 01:43:43.527376 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.527338 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64"] Sep 16 01:43:43.530495 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.530466 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.533412 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.533352 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8d230b\"/\"openshift-service-ca.crt\"" Sep 16 01:43:43.533412 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.533384 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8d230b\"/\"ipp-route-8d230b-kserve-self-signed-certs\"" Sep 16 01:43:43.533412 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.533396 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8d230b\"/\"kube-root-ca.crt\"" Sep 16 01:43:43.533685 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.533361 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8d230b\"/\"default-dockercfg-sjrr6\"" Sep 16 01:43:43.537820 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.537786 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:43.541200 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.541177 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64"] Sep 16 01:43:43.579696 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.579664 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-tmp-dir\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.579887 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.579718 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-home\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.579887 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.579741 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5038efde-3484-4cda-80c3-482e794daf28-tls-certs\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.579887 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.579766 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zhlc\" (UniqueName: \"kubernetes.io/projected/5038efde-3484-4cda-80c3-482e794daf28-kube-api-access-9zhlc\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.579887 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.579828 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-model-cache\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.579887 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.579864 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-dshm\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.681351 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.681311 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-dshm\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.681539 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.681383 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-tmp-dir\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.681539 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.681423 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-home\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.681539 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.681445 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5038efde-3484-4cda-80c3-482e794daf28-tls-certs\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.681703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.681566 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zhlc\" (UniqueName: \"kubernetes.io/projected/5038efde-3484-4cda-80c3-482e794daf28-kube-api-access-9zhlc\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.681703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.681649 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-model-cache\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.681846 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.681823 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-home\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.681925 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.681851 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-tmp-dir\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.682022 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.681994 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-model-cache\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.683769 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.683746 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-dshm\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.684336 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.684317 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5038efde-3484-4cda-80c3-482e794daf28-tls-certs\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.691316 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.691291 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zhlc\" (UniqueName: \"kubernetes.io/projected/5038efde-3484-4cda-80c3-482e794daf28-kube-api-access-9zhlc\") pod \"ipp-route-8d230b-kserve-864f44bc87-jlj64\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:43.844242 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:43.844142 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:44.126672 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:44.126642 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64"] Sep 16 01:43:44.127891 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:43:44.127863 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5038efde_3484_4cda_80c3_482e794daf28.slice/crio-acffa11160f375832dc9d0d3ee426d07a9320c4cdfbfd202804fa8fd010c87ae WatchSource:0}: Error finding container acffa11160f375832dc9d0d3ee426d07a9320c4cdfbfd202804fa8fd010c87ae: Status 404 returned error can't find the container with id acffa11160f375832dc9d0d3ee426d07a9320c4cdfbfd202804fa8fd010c87ae Sep 16 01:43:45.022111 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:45.022070 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:43:45.030316 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:45.030281 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" event={"ID":"5038efde-3484-4cda-80c3-482e794daf28","Type":"ContainerStarted","Data":"5cb713913d6de53d2571dd7fb415da8ab2b437e288ef4c1f3261a604be8cdc20"} Sep 16 01:43:45.030463 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:45.030327 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" event={"ID":"5038efde-3484-4cda-80c3-482e794daf28","Type":"ContainerStarted","Data":"acffa11160f375832dc9d0d3ee426d07a9320c4cdfbfd202804fa8fd010c87ae"} Sep 16 01:43:45.046852 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:45.046806 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" podStartSLOduration=2.046792333 podStartE2EDuration="2.046792333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:43:45.045369357 +0000 UTC m=+1655.297791945" watchObservedRunningTime="2026-09-16 01:43:45.046792333 +0000 UTC m=+1655.299214959" Sep 16 01:43:46.038611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:46.038580 2558 generic.go:356] "Generic (PLEG): container finished" podID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerID="461df808e25232900f5c87e59c8d5f53b99b1161a745f2fbd8a44ce67913f419" exitCode=0 Sep 16 01:43:46.039015 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:46.038657 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" event={"ID":"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a","Type":"ContainerDied","Data":"461df808e25232900f5c87e59c8d5f53b99b1161a745f2fbd8a44ce67913f419"} Sep 16 01:43:46.039209 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:46.039192 2558 scope.go:117] "RemoveContainer" containerID="461df808e25232900f5c87e59c8d5f53b99b1161a745f2fbd8a44ce67913f419" Sep 16 01:43:46.040501 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:46.040471 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:46.040587 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:46.040538 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:46.040587 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:46.040570 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:43:47.046215 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:47.046184 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" event={"ID":"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a","Type":"ContainerStarted","Data":"5f33a3c79494310b24286b128cb8882e01145dcb8e9d2b4f1bdc81bfe0a2317d"} Sep 16 01:43:47.743706 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:47.743674 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:47.744937 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:47.744904 2558 patch_prober.go:28] interesting pod/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 16 01:43:47.745061 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:47.744956 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 16 01:43:48.257858 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:48.257829 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:48.743876 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:48.743838 2558 patch_prober.go:28] interesting pod/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 16 01:43:48.744201 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:48.743906 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 16 01:43:49.386933 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:49.386895 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:49.743183 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:49.743071 2558 patch_prober.go:28] interesting pod/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 16 01:43:49.743183 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:49.743131 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 16 01:43:49.984842 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:49.984810 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:50.743081 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:50.743047 2558 patch_prober.go:28] interesting pod/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 16 01:43:50.743500 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:50.743103 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 16 01:43:50.769012 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:50.768982 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:50.850475 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:50.850444 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:51.408718 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:51.408682 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:51.743855 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:51.743770 2558 patch_prober.go:28] interesting pod/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 16 01:43:51.743855 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:51.743821 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 16 01:43:52.018888 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:52.018785 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:52.119316 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:52.119286 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:52.747094 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:52.747064 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:52.748124 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:52.748106 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:43:52.779430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:52.779405 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:53.658563 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:53.658534 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:53.844824 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:53.844786 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:53.845231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:53.844940 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:53.857325 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:53.857302 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:54.107225 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:54.107138 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:43:54.351801 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:54.351769 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:55.440077 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:55.440049 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:56.271298 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:56.271260 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:56.870368 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:56.870331 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:57.636625 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:57.636567 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:58.244236 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:58.244202 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:58.849636 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:58.849598 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:43:59.542858 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:43:59.542825 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:00.170273 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:00.170234 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:00.789662 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:00.789637 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:01.464060 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:01.464023 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:01.616380 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:01.616344 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:44:05.258233 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:05.258201 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:10.811896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:10.811863 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:10.846840 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:10.846804 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t"] Sep 16 01:44:10.847117 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:10.847069 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" podUID="5569bd81-5f3b-462d-860c-ebdc3d7814ff" containerName="payload-pre-processing" containerID="cri-o://ac24b2ef96c63dc152fc9af7d538ee4518a6ea84db5dd44628c9281eea497d7b" gracePeriod=30 Sep 16 01:44:10.912513 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:10.912482 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:10.991220 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:10.991189 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:11.046509 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.046469 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:11.112687 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.112665 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" Sep 16 01:44:11.164033 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.164003 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5569bd81-5f3b-462d-860c-ebdc3d7814ff-plugins-config-volume\") pod \"5569bd81-5f3b-462d-860c-ebdc3d7814ff\" (UID: \"5569bd81-5f3b-462d-860c-ebdc3d7814ff\") " Sep 16 01:44:11.164224 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.164106 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzj52\" (UniqueName: \"kubernetes.io/projected/5569bd81-5f3b-462d-860c-ebdc3d7814ff-kube-api-access-fzj52\") pod \"5569bd81-5f3b-462d-860c-ebdc3d7814ff\" (UID: \"5569bd81-5f3b-462d-860c-ebdc3d7814ff\") " Sep 16 01:44:11.164576 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.164397 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5569bd81-5f3b-462d-860c-ebdc3d7814ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5569bd81-5f3b-462d-860c-ebdc3d7814ff" (UID: "5569bd81-5f3b-462d-860c-ebdc3d7814ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:44:11.164576 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.164527 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5569bd81-5f3b-462d-860c-ebdc3d7814ff-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:11.171205 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.167701 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5569bd81-5f3b-462d-860c-ebdc3d7814ff-kube-api-access-fzj52" (OuterVolumeSpecName: "kube-api-access-fzj52") pod "5569bd81-5f3b-462d-860c-ebdc3d7814ff" (UID: "5569bd81-5f3b-462d-860c-ebdc3d7814ff"). InnerVolumeSpecName "kube-api-access-fzj52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:44:11.202933 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.202900 2558 generic.go:356] "Generic (PLEG): container finished" podID="5569bd81-5f3b-462d-860c-ebdc3d7814ff" containerID="ac24b2ef96c63dc152fc9af7d538ee4518a6ea84db5dd44628c9281eea497d7b" exitCode=0 Sep 16 01:44:11.203201 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.202953 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" Sep 16 01:44:11.203201 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.202985 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" event={"ID":"5569bd81-5f3b-462d-860c-ebdc3d7814ff","Type":"ContainerDied","Data":"ac24b2ef96c63dc152fc9af7d538ee4518a6ea84db5dd44628c9281eea497d7b"} Sep 16 01:44:11.203201 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.203042 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t" event={"ID":"5569bd81-5f3b-462d-860c-ebdc3d7814ff","Type":"ContainerDied","Data":"21f30ffd21c354292a7175f4d1088c89447d95614a0cd8fe0c6098053ec6dfb3"} Sep 16 01:44:11.203201 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.203072 2558 scope.go:117] "RemoveContainer" containerID="ac24b2ef96c63dc152fc9af7d538ee4518a6ea84db5dd44628c9281eea497d7b" Sep 16 01:44:11.214902 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.214879 2558 scope.go:117] "RemoveContainer" containerID="ac24b2ef96c63dc152fc9af7d538ee4518a6ea84db5dd44628c9281eea497d7b" Sep 16 01:44:11.215210 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:44:11.215189 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac24b2ef96c63dc152fc9af7d538ee4518a6ea84db5dd44628c9281eea497d7b\": container with ID starting with ac24b2ef96c63dc152fc9af7d538ee4518a6ea84db5dd44628c9281eea497d7b not found: ID does not exist" containerID="ac24b2ef96c63dc152fc9af7d538ee4518a6ea84db5dd44628c9281eea497d7b" Sep 16 01:44:11.215284 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.215221 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac24b2ef96c63dc152fc9af7d538ee4518a6ea84db5dd44628c9281eea497d7b"} err="failed to get container status \"ac24b2ef96c63dc152fc9af7d538ee4518a6ea84db5dd44628c9281eea497d7b\": rpc error: code = NotFound desc = could not find container \"ac24b2ef96c63dc152fc9af7d538ee4518a6ea84db5dd44628c9281eea497d7b\": container with ID starting with ac24b2ef96c63dc152fc9af7d538ee4518a6ea84db5dd44628c9281eea497d7b not found: ID does not exist" Sep 16 01:44:11.227851 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.227824 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t"] Sep 16 01:44:11.231487 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.231463 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d0959-68c54695c8-bqb7t"] Sep 16 01:44:11.266053 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.266029 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzj52\" (UniqueName: \"kubernetes.io/projected/5569bd81-5f3b-462d-860c-ebdc3d7814ff-kube-api-access-fzj52\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:11.418018 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.417990 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:11.518892 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:11.518855 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:12.057456 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:12.057413 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:12.295531 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:12.295498 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5569bd81-5f3b-462d-860c-ebdc3d7814ff" path="/var/lib/kubelet/pods/5569bd81-5f3b-462d-860c-ebdc3d7814ff/volumes" Sep 16 01:44:12.981445 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:12.981397 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:13.568317 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:13.568283 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:14.267253 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:14.267214 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:14.877645 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:14.877607 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:15.469811 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:15.469775 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:17.180497 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:17.180458 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:17.243501 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:17.243470 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:17.440741 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:17.440650 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42"] Sep 16 01:44:17.440936 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:17.440908 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" podUID="75cba283-9c47-467b-9ea1-c82250b6e394" containerName="istio-proxy" containerID="cri-o://91e657332d381836ae295b03d1b50653851f5753459f0bf16e1636d9324c4961" gracePeriod=30 Sep 16 01:44:17.463523 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:17.463495 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:17.970014 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:17.969968 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:18.705332 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:18.705303 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:19.419983 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:19.419951 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:44:19.455203 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:19.455172 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:20.165688 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:20.165653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:20.961807 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:20.961767 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:21.591691 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:21.591663 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:22.199764 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.199735 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:22.704595 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.704532 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:44:22.776320 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.776290 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75cba283-9c47-467b-9ea1-c82250b6e394-istio-token\") pod \"75cba283-9c47-467b-9ea1-c82250b6e394\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " Sep 16 01:44:22.776470 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.776338 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-istio-envoy\") pod \"75cba283-9c47-467b-9ea1-c82250b6e394\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " Sep 16 01:44:22.776470 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.776415 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cba283-9c47-467b-9ea1-c82250b6e394-istio-podinfo\") pod \"75cba283-9c47-467b-9ea1-c82250b6e394\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " Sep 16 01:44:22.776470 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.776457 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-workload-certs\") pod \"75cba283-9c47-467b-9ea1-c82250b6e394\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " Sep 16 01:44:22.776611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.776501 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4xqj\" (UniqueName: \"kubernetes.io/projected/75cba283-9c47-467b-9ea1-c82250b6e394-kube-api-access-z4xqj\") pod \"75cba283-9c47-467b-9ea1-c82250b6e394\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " Sep 16 01:44:22.776611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.776519 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-workload-socket\") pod \"75cba283-9c47-467b-9ea1-c82250b6e394\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " Sep 16 01:44:22.776611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.776543 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cba283-9c47-467b-9ea1-c82250b6e394-istiod-ca-cert\") pod \"75cba283-9c47-467b-9ea1-c82250b6e394\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " Sep 16 01:44:22.776611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.776569 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-istio-data\") pod \"75cba283-9c47-467b-9ea1-c82250b6e394\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " Sep 16 01:44:22.776611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.776599 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-credential-socket\") pod \"75cba283-9c47-467b-9ea1-c82250b6e394\" (UID: \"75cba283-9c47-467b-9ea1-c82250b6e394\") " Sep 16 01:44:22.776911 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.776884 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75cba283-9c47-467b-9ea1-c82250b6e394" (UID: "75cba283-9c47-467b-9ea1-c82250b6e394"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:44:22.776986 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.776892 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75cba283-9c47-467b-9ea1-c82250b6e394" (UID: "75cba283-9c47-467b-9ea1-c82250b6e394"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:44:22.777231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.777039 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-workload-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:22.777231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.777057 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-workload-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:22.777231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.777097 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-istio-data" (OuterVolumeSpecName: "istio-data") pod "75cba283-9c47-467b-9ea1-c82250b6e394" (UID: "75cba283-9c47-467b-9ea1-c82250b6e394"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:44:22.777231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.777138 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75cba283-9c47-467b-9ea1-c82250b6e394" (UID: "75cba283-9c47-467b-9ea1-c82250b6e394"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:44:22.777455 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.777307 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75cba283-9c47-467b-9ea1-c82250b6e394-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75cba283-9c47-467b-9ea1-c82250b6e394" (UID: "75cba283-9c47-467b-9ea1-c82250b6e394"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:44:22.778725 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.778700 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75cba283-9c47-467b-9ea1-c82250b6e394" (UID: "75cba283-9c47-467b-9ea1-c82250b6e394"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:44:22.778875 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.778853 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cba283-9c47-467b-9ea1-c82250b6e394-istio-token" (OuterVolumeSpecName: "istio-token") pod "75cba283-9c47-467b-9ea1-c82250b6e394" (UID: "75cba283-9c47-467b-9ea1-c82250b6e394"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:44:22.778983 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.778962 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cba283-9c47-467b-9ea1-c82250b6e394-kube-api-access-z4xqj" (OuterVolumeSpecName: "kube-api-access-z4xqj") pod "75cba283-9c47-467b-9ea1-c82250b6e394" (UID: "75cba283-9c47-467b-9ea1-c82250b6e394"). InnerVolumeSpecName "kube-api-access-z4xqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:44:22.779121 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.779106 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75cba283-9c47-467b-9ea1-c82250b6e394-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75cba283-9c47-467b-9ea1-c82250b6e394" (UID: "75cba283-9c47-467b-9ea1-c82250b6e394"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:44:22.878471 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.878443 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cba283-9c47-467b-9ea1-c82250b6e394-istio-podinfo\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:22.878471 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.878468 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4xqj\" (UniqueName: \"kubernetes.io/projected/75cba283-9c47-467b-9ea1-c82250b6e394-kube-api-access-z4xqj\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:22.878638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.878479 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cba283-9c47-467b-9ea1-c82250b6e394-istiod-ca-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:22.878638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.878487 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-istio-data\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:22.878638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.878495 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-credential-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:22.878638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.878503 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75cba283-9c47-467b-9ea1-c82250b6e394-istio-token\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:22.878638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.878511 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cba283-9c47-467b-9ea1-c82250b6e394-istio-envoy\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:22.976672 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:22.976599 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:23.292451 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:23.292376 2558 generic.go:356] "Generic (PLEG): container finished" podID="75cba283-9c47-467b-9ea1-c82250b6e394" containerID="91e657332d381836ae295b03d1b50653851f5753459f0bf16e1636d9324c4961" exitCode=0 Sep 16 01:44:23.292451 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:23.292431 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" Sep 16 01:44:23.292644 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:23.292465 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" event={"ID":"75cba283-9c47-467b-9ea1-c82250b6e394","Type":"ContainerDied","Data":"91e657332d381836ae295b03d1b50653851f5753459f0bf16e1636d9324c4961"} Sep 16 01:44:23.292644 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:23.292517 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42" event={"ID":"75cba283-9c47-467b-9ea1-c82250b6e394","Type":"ContainerDied","Data":"b2269b9f60209a25094a1ba250a9b785cb24baf87fe8b761c219b4ece5e88921"} Sep 16 01:44:23.292644 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:23.292539 2558 scope.go:117] "RemoveContainer" containerID="91e657332d381836ae295b03d1b50653851f5753459f0bf16e1636d9324c4961" Sep 16 01:44:23.302753 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:23.302727 2558 scope.go:117] "RemoveContainer" containerID="548118c5c9c236a6dca7621f13826cd64e61f41cef097029726e21f784d29c42" Sep 16 01:44:23.311392 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:23.311375 2558 scope.go:117] "RemoveContainer" containerID="91e657332d381836ae295b03d1b50653851f5753459f0bf16e1636d9324c4961" Sep 16 01:44:23.311653 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:44:23.311634 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e657332d381836ae295b03d1b50653851f5753459f0bf16e1636d9324c4961\": container with ID starting with 91e657332d381836ae295b03d1b50653851f5753459f0bf16e1636d9324c4961 not found: ID does not exist" containerID="91e657332d381836ae295b03d1b50653851f5753459f0bf16e1636d9324c4961" Sep 16 01:44:23.311700 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:23.311661 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e657332d381836ae295b03d1b50653851f5753459f0bf16e1636d9324c4961"} err="failed to get container status \"91e657332d381836ae295b03d1b50653851f5753459f0bf16e1636d9324c4961\": rpc error: code = NotFound desc = could not find container \"91e657332d381836ae295b03d1b50653851f5753459f0bf16e1636d9324c4961\": container with ID starting with 91e657332d381836ae295b03d1b50653851f5753459f0bf16e1636d9324c4961 not found: ID does not exist" Sep 16 01:44:23.311700 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:23.311677 2558 scope.go:117] "RemoveContainer" containerID="548118c5c9c236a6dca7621f13826cd64e61f41cef097029726e21f784d29c42" Sep 16 01:44:23.311956 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:44:23.311938 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"548118c5c9c236a6dca7621f13826cd64e61f41cef097029726e21f784d29c42\": container with ID starting with 548118c5c9c236a6dca7621f13826cd64e61f41cef097029726e21f784d29c42 not found: ID does not exist" containerID="548118c5c9c236a6dca7621f13826cd64e61f41cef097029726e21f784d29c42" Sep 16 01:44:23.311997 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:23.311963 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548118c5c9c236a6dca7621f13826cd64e61f41cef097029726e21f784d29c42"} err="failed to get container status \"548118c5c9c236a6dca7621f13826cd64e61f41cef097029726e21f784d29c42\": rpc error: code = NotFound desc = could not find container \"548118c5c9c236a6dca7621f13826cd64e61f41cef097029726e21f784d29c42\": container with ID starting with 548118c5c9c236a6dca7621f13826cd64e61f41cef097029726e21f784d29c42 not found: ID does not exist" Sep 16 01:44:23.316559 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:23.316527 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42"] Sep 16 01:44:23.319651 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:23.319630 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7d0959-openshift-default-6cc9b9ddf6-z9v42"] Sep 16 01:44:24.295801 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:24.295758 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75cba283-9c47-467b-9ea1-c82250b6e394" path="/var/lib/kubelet/pods/75cba283-9c47-467b-9ea1-c82250b6e394/volumes" Sep 16 01:44:26.433642 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:26.433612 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5_ab5a3df4-818f-4c84-8187-c29a4ca5b7e3/payload-processing/0.log" Sep 16 01:44:26.554107 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:26.554084 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5_ab5a3df4-818f-4c84-8187-c29a4ca5b7e3/payload-processing/0.log" Sep 16 01:44:26.826191 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:26.826105 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:27.131243 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.131206 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7"] Sep 16 01:44:27.131871 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.131855 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75cba283-9c47-467b-9ea1-c82250b6e394" containerName="istio-proxy" Sep 16 01:44:27.131916 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.131875 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="75cba283-9c47-467b-9ea1-c82250b6e394" containerName="istio-proxy" Sep 16 01:44:27.131916 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.131896 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5569bd81-5f3b-462d-860c-ebdc3d7814ff" containerName="payload-pre-processing" Sep 16 01:44:27.131916 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.131904 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5569bd81-5f3b-462d-860c-ebdc3d7814ff" containerName="payload-pre-processing" Sep 16 01:44:27.132061 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.132048 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="75cba283-9c47-467b-9ea1-c82250b6e394" containerName="istio-proxy" Sep 16 01:44:27.132109 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.132071 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5569bd81-5f3b-462d-860c-ebdc3d7814ff" containerName="payload-pre-processing" Sep 16 01:44:27.137250 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.137224 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.139517 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.139471 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-f73a36-openshift-default-dockercfg-q7pgc\"" Sep 16 01:44:27.143988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.143963 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7"] Sep 16 01:44:27.159575 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.159550 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:27.218415 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.218388 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-workload-socket\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.218556 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.218427 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-workload-certs\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.218556 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.218493 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-credential-socket\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.218556 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.218529 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27g6c\" (UniqueName: \"kubernetes.io/projected/d32f1343-9c5c-4333-ba98-fbd9c80655c6-kube-api-access-27g6c\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.218688 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.218601 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-podinfo\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.218688 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.218633 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.218688 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.218663 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-token\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.218808 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.218701 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-envoy\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.218808 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.218754 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-data\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.319447 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.319414 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-data\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.319615 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.319468 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-workload-socket\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.319615 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.319579 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-workload-certs\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.319703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.319625 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-credential-socket\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.319703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.319651 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27g6c\" (UniqueName: \"kubernetes.io/projected/d32f1343-9c5c-4333-ba98-fbd9c80655c6-kube-api-access-27g6c\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.319703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.319700 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-podinfo\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.319859 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.319720 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.319859 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.319782 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-token\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.319859 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.319803 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-workload-socket\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.319859 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.319835 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-envoy\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.319859 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.319835 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-data\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.320098 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.319863 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-workload-certs\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.320098 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.319903 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-credential-socket\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.320354 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.320335 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.322249 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.322227 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-envoy\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.322362 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.322343 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-podinfo\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.327443 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.327419 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-token\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.327621 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.327601 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27g6c\" (UniqueName: \"kubernetes.io/projected/d32f1343-9c5c-4333-ba98-fbd9c80655c6-kube-api-access-27g6c\") pod \"e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.385920 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.385838 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:27.450234 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.450209 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:27.735641 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.735615 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7"] Sep 16 01:44:27.737001 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:44:27.736967 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32f1343_9c5c_4333_ba98_fbd9c80655c6.slice/crio-313464a891db6d3a91845208a9af49c9cbe7507017eaa4967ff574505e452bd9 WatchSource:0}: Error finding container 313464a891db6d3a91845208a9af49c9cbe7507017eaa4967ff574505e452bd9: Status 404 returned error can't find the container with id 313464a891db6d3a91845208a9af49c9cbe7507017eaa4967ff574505e452bd9 Sep 16 01:44:27.739292 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.739255 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:44:27.739407 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.739326 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:44:27.739407 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:27.739354 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:44:28.323634 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:28.323601 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" event={"ID":"d32f1343-9c5c-4333-ba98-fbd9c80655c6","Type":"ContainerStarted","Data":"3b1a4aa58765cf3f5a618e24094b7459957983145ae0cb5d610f9eae1f74489a"} Sep 16 01:44:28.323830 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:28.323648 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" event={"ID":"d32f1343-9c5c-4333-ba98-fbd9c80655c6","Type":"ContainerStarted","Data":"313464a891db6d3a91845208a9af49c9cbe7507017eaa4967ff574505e452bd9"} Sep 16 01:44:28.343556 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:28.343507 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" podStartSLOduration=1.343492031 podStartE2EDuration="1.343492031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:44:28.340547001 +0000 UTC m=+1698.592969593" watchObservedRunningTime="2026-09-16 01:44:28.343492031 +0000 UTC m=+1698.595914619" Sep 16 01:44:28.453096 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:28.453066 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:28.453478 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:28.453201 2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Sep 16 01:44:28.453478 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:28.453243 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Sep 16 01:44:29.456003 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:29.455970 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:29.660034 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:29.659990 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:30.279720 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:30.279685 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:30.337613 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:30.337569 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:30.338830 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:30.338807 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:30.363387 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:30.363350 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:30.455137 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:30.455104 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:30.615628 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:30.615595 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:30.714962 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:30.714929 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:30.817188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:30.817122 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:30.913663 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:30.913565 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:31.110347 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:31.110310 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:31.661679 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:31.661643 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:32.267873 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:32.267845 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:33.075794 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:33.075769 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:33.689581 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:33.689541 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:34.479402 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:34.479355 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:34.533367 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:34.533334 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:34.720230 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:34.720197 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:34.917990 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:34.917962 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:35.017664 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:35.017602 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:35.117822 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:35.117792 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:44:35.288015 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:35.287928 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:35.898517 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:35.898488 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:36.622372 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:36.622334 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:37.438150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:37.438118 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:37.500063 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:37.500035 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:37.685593 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:37.685560 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb"] Sep 16 01:44:37.685882 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:37.685853 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerName="istio-proxy" containerID="cri-o://5f33a3c79494310b24286b128cb8882e01145dcb8e9d2b4f1bdc81bfe0a2317d" gracePeriod=30 Sep 16 01:44:37.704353 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:37.704283 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:37.750131 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:37.750102 2558 patch_prober.go:28] interesting pod/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 01:44:37.750272 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:37.750191 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 01:44:38.362105 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:38.362068 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:39.264093 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:39.264046 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:39.857263 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:39.857231 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:40.465389 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:40.465361 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:41.263454 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:41.263419 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:41.859174 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:41.859133 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:42.564875 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:42.564834 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:42.958407 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:42.958384 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:44:43.074691 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.074655 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-podinfo\") pod \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " Sep 16 01:44:43.074691 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.074695 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6k6f\" (UniqueName: \"kubernetes.io/projected/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-kube-api-access-k6k6f\") pod \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " Sep 16 01:44:43.074894 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.074715 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-workload-certs\") pod \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " Sep 16 01:44:43.074894 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.074775 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istiod-ca-cert\") pod \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " Sep 16 01:44:43.074894 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.074831 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-envoy\") pod \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " Sep 16 01:44:43.074894 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.074862 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-workload-socket\") pod \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " Sep 16 01:44:43.074894 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.074882 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-token\") pod \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " Sep 16 01:44:43.075191 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.074910 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-credential-socket\") pod \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " Sep 16 01:44:43.075191 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.074947 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-data\") pod \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\" (UID: \"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a\") " Sep 16 01:44:43.075191 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.075101 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" (UID: "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:44:43.075351 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.075294 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" (UID: "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:44:43.075424 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.075400 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-data" (OuterVolumeSpecName: "istio-data") pod "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" (UID: "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:44:43.075486 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.075420 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-workload-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:43.075486 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.075423 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" (UID: "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:44:43.075772 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.075747 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" (UID: "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:44:43.077273 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.077246 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" (UID: "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:44:43.077389 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.077290 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" (UID: "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:44:43.077389 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.077341 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-kube-api-access-k6k6f" (OuterVolumeSpecName: "kube-api-access-k6k6f") pod "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" (UID: "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a"). InnerVolumeSpecName "kube-api-access-k6k6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:44:43.077536 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.077519 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-token" (OuterVolumeSpecName: "istio-token") pod "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" (UID: "b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:44:43.173048 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.173018 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:43.176227 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.176201 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-data\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:43.176227 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.176227 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-podinfo\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:43.176405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.176241 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6k6f\" (UniqueName: \"kubernetes.io/projected/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-kube-api-access-k6k6f\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:43.176405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.176255 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istiod-ca-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:43.176405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.176275 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-envoy\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:43.176405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.176286 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-workload-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:43.176405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.176296 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-istio-token\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:43.176405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.176308 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a-credential-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:43.417752 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.417719 2558 generic.go:356] "Generic (PLEG): container finished" podID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerID="5f33a3c79494310b24286b128cb8882e01145dcb8e9d2b4f1bdc81bfe0a2317d" exitCode=0 Sep 16 01:44:43.417916 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.417764 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" Sep 16 01:44:43.417916 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.417805 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" event={"ID":"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a","Type":"ContainerDied","Data":"5f33a3c79494310b24286b128cb8882e01145dcb8e9d2b4f1bdc81bfe0a2317d"} Sep 16 01:44:43.417916 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.417859 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb" event={"ID":"b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a","Type":"ContainerDied","Data":"05f51fccfdf30f1015f2d1459306725229d12c72dd44059f9fc48fae5b547438"} Sep 16 01:44:43.417916 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.417881 2558 scope.go:117] "RemoveContainer" containerID="5f33a3c79494310b24286b128cb8882e01145dcb8e9d2b4f1bdc81bfe0a2317d" Sep 16 01:44:43.428728 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.428711 2558 scope.go:117] "RemoveContainer" containerID="461df808e25232900f5c87e59c8d5f53b99b1161a745f2fbd8a44ce67913f419" Sep 16 01:44:43.438781 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.438760 2558 scope.go:117] "RemoveContainer" containerID="5f33a3c79494310b24286b128cb8882e01145dcb8e9d2b4f1bdc81bfe0a2317d" Sep 16 01:44:43.439080 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:44:43.439061 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f33a3c79494310b24286b128cb8882e01145dcb8e9d2b4f1bdc81bfe0a2317d\": container with ID starting with 5f33a3c79494310b24286b128cb8882e01145dcb8e9d2b4f1bdc81bfe0a2317d not found: ID does not exist" containerID="5f33a3c79494310b24286b128cb8882e01145dcb8e9d2b4f1bdc81bfe0a2317d" Sep 16 01:44:43.439184 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.439087 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f33a3c79494310b24286b128cb8882e01145dcb8e9d2b4f1bdc81bfe0a2317d"} err="failed to get container status \"5f33a3c79494310b24286b128cb8882e01145dcb8e9d2b4f1bdc81bfe0a2317d\": rpc error: code = NotFound desc = could not find container \"5f33a3c79494310b24286b128cb8882e01145dcb8e9d2b4f1bdc81bfe0a2317d\": container with ID starting with 5f33a3c79494310b24286b128cb8882e01145dcb8e9d2b4f1bdc81bfe0a2317d not found: ID does not exist" Sep 16 01:44:43.439184 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.439103 2558 scope.go:117] "RemoveContainer" containerID="461df808e25232900f5c87e59c8d5f53b99b1161a745f2fbd8a44ce67913f419" Sep 16 01:44:43.439383 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:44:43.439361 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461df808e25232900f5c87e59c8d5f53b99b1161a745f2fbd8a44ce67913f419\": container with ID starting with 461df808e25232900f5c87e59c8d5f53b99b1161a745f2fbd8a44ce67913f419 not found: ID does not exist" containerID="461df808e25232900f5c87e59c8d5f53b99b1161a745f2fbd8a44ce67913f419" Sep 16 01:44:43.439447 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.439386 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461df808e25232900f5c87e59c8d5f53b99b1161a745f2fbd8a44ce67913f419"} err="failed to get container status \"461df808e25232900f5c87e59c8d5f53b99b1161a745f2fbd8a44ce67913f419\": rpc error: code = NotFound desc = could not find container \"461df808e25232900f5c87e59c8d5f53b99b1161a745f2fbd8a44ce67913f419\": container with ID starting with 461df808e25232900f5c87e59c8d5f53b99b1161a745f2fbd8a44ce67913f419 not found: ID does not exist" Sep 16 01:44:43.441179 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.441137 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb"] Sep 16 01:44:43.443842 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.443822 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4e1413-openshift-default-d485ff5f8-lv6vb"] Sep 16 01:44:43.831403 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:43.831306 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:44.296575 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:44.296542 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" path="/var/lib/kubelet/pods/b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a/volumes" Sep 16 01:44:44.452188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:44.452133 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:45.093951 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:45.093919 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:45.319015 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:45.318978 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:46.048488 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:46.048457 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:46.879390 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:46.879336 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:47.483616 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:47.483580 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:48.259566 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:48.259537 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:48.795193 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:48.788512 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:48.812727 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:48.812694 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt"] Sep 16 01:44:48.812958 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:48.812929 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" podUID="aa6eeecb-cd72-4ce8-82a3-96026d80131f" containerName="payload-processing" containerID="cri-o://bb22a594fe4e63df4b2aa191553cc1b8f6e7394119cb36bcdfe5b2540fe1b09a" gracePeriod=30 Sep 16 01:44:48.887031 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:48.886796 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:48.962013 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:48.961947 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:49.018910 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.018857 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:49.129819 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.129791 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" Sep 16 01:44:49.236719 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.236685 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cwwh\" (UniqueName: \"kubernetes.io/projected/aa6eeecb-cd72-4ce8-82a3-96026d80131f-kube-api-access-8cwwh\") pod \"aa6eeecb-cd72-4ce8-82a3-96026d80131f\" (UID: \"aa6eeecb-cd72-4ce8-82a3-96026d80131f\") " Sep 16 01:44:49.236904 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.236731 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6eeecb-cd72-4ce8-82a3-96026d80131f-plugins-config-volume\") pod \"aa6eeecb-cd72-4ce8-82a3-96026d80131f\" (UID: \"aa6eeecb-cd72-4ce8-82a3-96026d80131f\") " Sep 16 01:44:49.237137 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.237111 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6eeecb-cd72-4ce8-82a3-96026d80131f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa6eeecb-cd72-4ce8-82a3-96026d80131f" (UID: "aa6eeecb-cd72-4ce8-82a3-96026d80131f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:44:49.238839 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.238814 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6eeecb-cd72-4ce8-82a3-96026d80131f-kube-api-access-8cwwh" (OuterVolumeSpecName: "kube-api-access-8cwwh") pod "aa6eeecb-cd72-4ce8-82a3-96026d80131f" (UID: "aa6eeecb-cd72-4ce8-82a3-96026d80131f"). InnerVolumeSpecName "kube-api-access-8cwwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:44:49.335301 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.335267 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:49.338502 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.338482 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cwwh\" (UniqueName: \"kubernetes.io/projected/aa6eeecb-cd72-4ce8-82a3-96026d80131f-kube-api-access-8cwwh\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:49.338576 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.338505 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6eeecb-cd72-4ce8-82a3-96026d80131f-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:44:49.464087 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.464050 2558 generic.go:356] "Generic (PLEG): container finished" podID="aa6eeecb-cd72-4ce8-82a3-96026d80131f" containerID="bb22a594fe4e63df4b2aa191553cc1b8f6e7394119cb36bcdfe5b2540fe1b09a" exitCode=0 Sep 16 01:44:49.464288 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.464133 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" event={"ID":"aa6eeecb-cd72-4ce8-82a3-96026d80131f","Type":"ContainerDied","Data":"bb22a594fe4e63df4b2aa191553cc1b8f6e7394119cb36bcdfe5b2540fe1b09a"} Sep 16 01:44:49.464288 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.464216 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" event={"ID":"aa6eeecb-cd72-4ce8-82a3-96026d80131f","Type":"ContainerDied","Data":"1a4c7a8b170a2465e40a4808de97aca164f53e9974a2639208a1241bc25ebc67"} Sep 16 01:44:49.464288 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.464261 2558 scope.go:117] "RemoveContainer" containerID="bb22a594fe4e63df4b2aa191553cc1b8f6e7394119cb36bcdfe5b2540fe1b09a" Sep 16 01:44:49.464505 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.464394 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt" Sep 16 01:44:49.487948 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.487228 2558 scope.go:117] "RemoveContainer" containerID="bb22a594fe4e63df4b2aa191553cc1b8f6e7394119cb36bcdfe5b2540fe1b09a" Sep 16 01:44:49.487948 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.487669 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:49.489196 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:44:49.488524 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb22a594fe4e63df4b2aa191553cc1b8f6e7394119cb36bcdfe5b2540fe1b09a\": container with ID starting with bb22a594fe4e63df4b2aa191553cc1b8f6e7394119cb36bcdfe5b2540fe1b09a not found: ID does not exist" containerID="bb22a594fe4e63df4b2aa191553cc1b8f6e7394119cb36bcdfe5b2540fe1b09a" Sep 16 01:44:49.489196 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.488564 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb22a594fe4e63df4b2aa191553cc1b8f6e7394119cb36bcdfe5b2540fe1b09a"} err="failed to get container status \"bb22a594fe4e63df4b2aa191553cc1b8f6e7394119cb36bcdfe5b2540fe1b09a\": rpc error: code = NotFound desc = could not find container \"bb22a594fe4e63df4b2aa191553cc1b8f6e7394119cb36bcdfe5b2540fe1b09a\": container with ID starting with bb22a594fe4e63df4b2aa191553cc1b8f6e7394119cb36bcdfe5b2540fe1b09a not found: ID does not exist" Sep 16 01:44:49.498980 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.498957 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt"] Sep 16 01:44:49.504117 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:49.504096 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4bc0fb-59475f7f7c-jmljt"] Sep 16 01:44:50.103499 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:50.103465 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:50.296442 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:50.296410 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa6eeecb-cd72-4ce8-82a3-96026d80131f" path="/var/lib/kubelet/pods/aa6eeecb-cd72-4ce8-82a3-96026d80131f/volumes" Sep 16 01:44:50.856520 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:50.856486 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:51.630571 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:51.630536 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:52.505638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:52.505599 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:53.067208 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:53.067144 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:53.979378 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:53.979342 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:54.684523 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:54.684482 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:55.300614 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:55.300580 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:55.513364 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:55.513329 2558 generic.go:356] "Generic (PLEG): container finished" podID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerID="3b1a4aa58765cf3f5a618e24094b7459957983145ae0cb5d610f9eae1f74489a" exitCode=0 Sep 16 01:44:55.513543 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:55.513386 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" event={"ID":"d32f1343-9c5c-4333-ba98-fbd9c80655c6","Type":"ContainerDied","Data":"3b1a4aa58765cf3f5a618e24094b7459957983145ae0cb5d610f9eae1f74489a"} Sep 16 01:44:55.513831 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:55.513813 2558 scope.go:117] "RemoveContainer" containerID="3b1a4aa58765cf3f5a618e24094b7459957983145ae0cb5d610f9eae1f74489a" Sep 16 01:44:55.515669 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:55.515632 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:44:55.515799 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:55.515726 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:44:55.515799 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:55.515775 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 16 01:44:55.520363 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:55.519688 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:44:55.895973 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:55.895941 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:55.951259 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:55.951231 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:56.074718 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:56.074688 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:56.181561 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:56.181480 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:56.522502 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:56.522424 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" event={"ID":"d32f1343-9c5c-4333-ba98-fbd9c80655c6","Type":"ContainerStarted","Data":"c443adcfc0ba8c7384ce6b19fb5d72abe982c4f2f602bd75730dcc796714ce3a"} Sep 16 01:44:56.951415 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:56.951379 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:57.450792 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:57.450762 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:44:57.452036 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:57.452010 2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Sep 16 01:44:57.452143 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:57.452059 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Sep 16 01:44:57.752209 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:57.752096 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:58.379609 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:58.379557 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:58.450671 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:58.450635 2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Sep 16 01:44:58.450849 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:58.450694 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Sep 16 01:44:59.222450 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:59.222418 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:44:59.451045 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:59.451004 2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Sep 16 01:44:59.451260 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:59.451063 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Sep 16 01:44:59.837213 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:44:59.837172 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:00.136077 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.136031 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp"] Sep 16 01:45:00.136944 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.136916 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerName="istio-proxy" Sep 16 01:45:00.137062 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.136947 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerName="istio-proxy" Sep 16 01:45:00.137062 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.136966 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa6eeecb-cd72-4ce8-82a3-96026d80131f" containerName="payload-processing" Sep 16 01:45:00.137062 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.136976 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa6eeecb-cd72-4ce8-82a3-96026d80131f" containerName="payload-processing" Sep 16 01:45:00.137062 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.136999 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75cba283-9c47-467b-9ea1-c82250b6e394" containerName="istio-proxy" Sep 16 01:45:00.137062 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.137009 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="75cba283-9c47-467b-9ea1-c82250b6e394" containerName="istio-proxy" Sep 16 01:45:00.137263 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.137111 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="75cba283-9c47-467b-9ea1-c82250b6e394" containerName="istio-proxy" Sep 16 01:45:00.137263 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.137121 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerName="istio-proxy" Sep 16 01:45:00.137263 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.137129 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa6eeecb-cd72-4ce8-82a3-96026d80131f" containerName="payload-processing" Sep 16 01:45:00.137263 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.137135 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerName="istio-proxy" Sep 16 01:45:00.139650 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.139628 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn"] Sep 16 01:45:00.139793 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.139770 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp" Sep 16 01:45:00.140264 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.140242 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerName="istio-proxy" Sep 16 01:45:00.140264 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.140264 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5ab75d0-b5a4-44c5-b74d-8246e7ca5c4a" containerName="istio-proxy" Sep 16 01:45:00.142292 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.142271 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4ljjd\"" Sep 16 01:45:00.143034 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.143015 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825385-t96t6"] Sep 16 01:45:00.143136 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.143122 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn" Sep 16 01:45:00.146105 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.146080 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825385-t96t6" Sep 16 01:45:00.153177 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.152237 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn"] Sep 16 01:45:00.154917 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.154882 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825385-t96t6"] Sep 16 01:45:00.157616 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.157590 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp" Sep 16 01:45:00.158449 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.158424 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp"] Sep 16 01:45:00.196439 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.196410 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn" Sep 16 01:45:00.208122 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.208097 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825385-t96t6" Sep 16 01:45:00.337978 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.337415 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:45:00.451821 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.451742 2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Sep 16 01:45:00.451821 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.451801 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Sep 16 01:45:00.480185 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.478615 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:00.522498 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.522470 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn"] Sep 16 01:45:00.524282 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:45:00.524255 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c553b16_a983_46b2_ba04_a19cccebc22f.slice/crio-faab9f5ef04624a474ee404dea26a1e1297ecde76c7edd3d94c273e7eb803bce WatchSource:0}: Error finding container faab9f5ef04624a474ee404dea26a1e1297ecde76c7edd3d94c273e7eb803bce: Status 404 returned error can't find the container with id faab9f5ef04624a474ee404dea26a1e1297ecde76c7edd3d94c273e7eb803bce Sep 16 01:45:00.525189 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:45:00.525134 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125ca277_f2a2_4d69_ab6c_853b1dc09588.slice/crio-412c953e3dcf4f8fdd4ee5c7d44cb8ddc43dfcfbedc33ae83d5b91c297265408 WatchSource:0}: Error finding container 412c953e3dcf4f8fdd4ee5c7d44cb8ddc43dfcfbedc33ae83d5b91c297265408: Status 404 returned error can't find the container with id 412c953e3dcf4f8fdd4ee5c7d44cb8ddc43dfcfbedc33ae83d5b91c297265408 Sep 16 01:45:00.525310 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.525280 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp"] Sep 16 01:45:00.535300 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.535279 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825385-t96t6"] Sep 16 01:45:00.538946 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:45:00.538922 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00be5b2f_36e7_4eb1_8248_d722ca0e72ac.slice/crio-0c757354b9be817224e0426e157a23b499baa9e1861603ea6644d696c27d2fd5 WatchSource:0}: Error finding container 0c757354b9be817224e0426e157a23b499baa9e1861603ea6644d696c27d2fd5: Status 404 returned error can't find the container with id 0c757354b9be817224e0426e157a23b499baa9e1861603ea6644d696c27d2fd5 Sep 16 01:45:00.553530 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.553446 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825385-t96t6" event={"ID":"00be5b2f-36e7-4eb1-8248-d722ca0e72ac","Type":"ContainerStarted","Data":"0c757354b9be817224e0426e157a23b499baa9e1861603ea6644d696c27d2fd5"} Sep 16 01:45:00.555929 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.555904 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn" event={"ID":"6c553b16-a983-46b2-ba04-a19cccebc22f","Type":"ContainerStarted","Data":"faab9f5ef04624a474ee404dea26a1e1297ecde76c7edd3d94c273e7eb803bce"} Sep 16 01:45:00.557430 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:00.557402 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp" event={"ID":"125ca277-f2a2-4d69-ab6c-853b1dc09588","Type":"ContainerStarted","Data":"412c953e3dcf4f8fdd4ee5c7d44cb8ddc43dfcfbedc33ae83d5b91c297265408"} Sep 16 01:45:01.265423 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:01.265303 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:01.451386 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:01.451347 2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Sep 16 01:45:01.451782 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:01.451418 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Sep 16 01:45:01.600301 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:01.600173 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825385-t96t6" event={"ID":"00be5b2f-36e7-4eb1-8248-d722ca0e72ac","Type":"ContainerStarted","Data":"5caa9e4908e62a503c3ceaedda935448de08cb86aba7c62d4cd500d987b0b560"} Sep 16 01:45:01.601891 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:01.601855 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn" event={"ID":"6c553b16-a983-46b2-ba04-a19cccebc22f","Type":"ContainerStarted","Data":"68c963fdfaa231fad18b38bec8c80ec6e5300687af436655ab2b2f571ba576ab"} Sep 16 01:45:01.603650 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:01.603621 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp" event={"ID":"125ca277-f2a2-4d69-ab6c-853b1dc09588","Type":"ContainerStarted","Data":"0b79b8d4df046793d8837b239dc1609de3c41aeb56ff1507d18262f6700dfe85"} Sep 16 01:45:01.613641 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:01.613598 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825385-t96t6" podStartSLOduration=1.613585099 podStartE2EDuration="1.613585099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:45:01.612587565 +0000 UTC m=+1731.865010154" watchObservedRunningTime="2026-09-16 01:45:01.613585099 +0000 UTC m=+1731.866007705" Sep 16 01:45:01.627833 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:01.627792 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn" podStartSLOduration=1.6277801090000001 podStartE2EDuration="1.627780109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:45:01.625119856 +0000 UTC m=+1731.877542457" watchObservedRunningTime="2026-09-16 01:45:01.627780109 +0000 UTC m=+1731.880202698" Sep 16 01:45:01.639491 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:01.639444 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp" podStartSLOduration=1.6394320119999999 podStartE2EDuration="1.639432012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:45:01.636697847 +0000 UTC m=+1731.889120460" watchObservedRunningTime="2026-09-16 01:45:01.639432012 +0000 UTC m=+1731.891854601" Sep 16 01:45:02.047749 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:02.047717 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:02.456524 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:02.456489 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:45:02.458308 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:02.458278 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:45:02.483240 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:02.483203 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:02.552199 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:02.552151 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:02.614498 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:02.614466 2558 generic.go:356] "Generic (PLEG): container finished" podID="00be5b2f-36e7-4eb1-8248-d722ca0e72ac" containerID="5caa9e4908e62a503c3ceaedda935448de08cb86aba7c62d4cd500d987b0b560" exitCode=0 Sep 16 01:45:02.614675 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:02.614547 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825385-t96t6" event={"ID":"00be5b2f-36e7-4eb1-8248-d722ca0e72ac","Type":"ContainerDied","Data":"5caa9e4908e62a503c3ceaedda935448de08cb86aba7c62d4cd500d987b0b560"} Sep 16 01:45:02.616420 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:02.616396 2558 generic.go:356] "Generic (PLEG): container finished" podID="6c553b16-a983-46b2-ba04-a19cccebc22f" containerID="68c963fdfaa231fad18b38bec8c80ec6e5300687af436655ab2b2f571ba576ab" exitCode=0 Sep 16 01:45:02.616547 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:02.616462 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn" event={"ID":"6c553b16-a983-46b2-ba04-a19cccebc22f","Type":"ContainerDied","Data":"68c963fdfaa231fad18b38bec8c80ec6e5300687af436655ab2b2f571ba576ab"} Sep 16 01:45:02.618068 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:02.618045 2558 generic.go:356] "Generic (PLEG): container finished" podID="125ca277-f2a2-4d69-ab6c-853b1dc09588" containerID="0b79b8d4df046793d8837b239dc1609de3c41aeb56ff1507d18262f6700dfe85" exitCode=0 Sep 16 01:45:02.618179 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:02.618106 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp" event={"ID":"125ca277-f2a2-4d69-ab6c-853b1dc09588","Type":"ContainerDied","Data":"0b79b8d4df046793d8837b239dc1609de3c41aeb56ff1507d18262f6700dfe85"} Sep 16 01:45:02.636666 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:02.636647 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:02.697543 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:02.697516 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:03.080406 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:03.080369 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:03.181123 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:03.181091 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:03.779902 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:03.779877 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825385-t96t6" Sep 16 01:45:03.813470 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:03.813447 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp" Sep 16 01:45:03.826654 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:03.826634 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn" Sep 16 01:45:03.980714 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:03.980641 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:04.631463 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:04.631432 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825385-t96t6" Sep 16 01:45:04.631628 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:04.631460 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825385-t96t6" event={"ID":"00be5b2f-36e7-4eb1-8248-d722ca0e72ac","Type":"ContainerDied","Data":"0c757354b9be817224e0426e157a23b499baa9e1861603ea6644d696c27d2fd5"} Sep 16 01:45:04.631628 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:04.631512 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c757354b9be817224e0426e157a23b499baa9e1861603ea6644d696c27d2fd5" Sep 16 01:45:04.632868 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:04.632846 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn" Sep 16 01:45:04.633012 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:04.632891 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn" event={"ID":"6c553b16-a983-46b2-ba04-a19cccebc22f","Type":"ContainerDied","Data":"faab9f5ef04624a474ee404dea26a1e1297ecde76c7edd3d94c273e7eb803bce"} Sep 16 01:45:04.633012 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:04.632935 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faab9f5ef04624a474ee404dea26a1e1297ecde76c7edd3d94c273e7eb803bce" Sep 16 01:45:04.634553 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:04.634535 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp" Sep 16 01:45:04.634661 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:04.634576 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp" event={"ID":"125ca277-f2a2-4d69-ab6c-853b1dc09588","Type":"ContainerDied","Data":"412c953e3dcf4f8fdd4ee5c7d44cb8ddc43dfcfbedc33ae83d5b91c297265408"} Sep 16 01:45:04.634661 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:04.634611 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="412c953e3dcf4f8fdd4ee5c7d44cb8ddc43dfcfbedc33ae83d5b91c297265408" Sep 16 01:45:04.753317 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:04.753286 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:05.398057 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:05.398023 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:06.270558 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:06.270524 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:06.810616 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:06.810588 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:06.874108 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:06.874069 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:06.973611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:06.973576 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:07.047806 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:07.047774 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:07.359413 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:07.359373 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:07.964885 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:07.964855 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:08.694529 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:08.694492 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:09.246292 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:09.246257 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:09.347023 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:09.346993 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:09.487188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:09.487133 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:09.504854 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:09.504784 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7"] Sep 16 01:45:09.505044 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:09.505021 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerName="istio-proxy" containerID="cri-o://c443adcfc0ba8c7384ce6b19fb5d72abe982c4f2f602bd75730dcc796714ce3a" gracePeriod=30 Sep 16 01:45:09.584948 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:09.584913 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:12.719706 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:12.719669 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:45:12.947597 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:12.947568 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:13.044309 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:13.044215 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:13.262447 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:13.262408 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:13.646190 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:13.646148 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:14.268406 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.268377 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:14.702169 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.702114 2558 generic.go:356] "Generic (PLEG): container finished" podID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerID="c443adcfc0ba8c7384ce6b19fb5d72abe982c4f2f602bd75730dcc796714ce3a" exitCode=0 Sep 16 01:45:14.702317 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.702201 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" event={"ID":"d32f1343-9c5c-4333-ba98-fbd9c80655c6","Type":"ContainerDied","Data":"c443adcfc0ba8c7384ce6b19fb5d72abe982c4f2f602bd75730dcc796714ce3a"} Sep 16 01:45:14.702317 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.702269 2558 scope.go:117] "RemoveContainer" containerID="3b1a4aa58765cf3f5a618e24094b7459957983145ae0cb5d610f9eae1f74489a" Sep 16 01:45:14.770601 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.770577 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:45:14.893269 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893238 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-workload-certs\") pod \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " Sep 16 01:45:14.893445 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893289 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-podinfo\") pod \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " Sep 16 01:45:14.893445 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893326 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-token\") pod \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " Sep 16 01:45:14.893445 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893410 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-workload-socket\") pod \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " Sep 16 01:45:14.893612 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893446 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-envoy\") pod \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " Sep 16 01:45:14.893612 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893493 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istiod-ca-cert\") pod \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " Sep 16 01:45:14.893612 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893537 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d32f1343-9c5c-4333-ba98-fbd9c80655c6" (UID: "d32f1343-9c5c-4333-ba98-fbd9c80655c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:45:14.893612 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893560 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-data\") pod \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " Sep 16 01:45:14.893612 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893597 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27g6c\" (UniqueName: \"kubernetes.io/projected/d32f1343-9c5c-4333-ba98-fbd9c80655c6-kube-api-access-27g6c\") pod \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " Sep 16 01:45:14.893860 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893639 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-credential-socket\") pod \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\" (UID: \"d32f1343-9c5c-4333-ba98-fbd9c80655c6\") " Sep 16 01:45:14.893860 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893837 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "d32f1343-9c5c-4333-ba98-fbd9c80655c6" (UID: "d32f1343-9c5c-4333-ba98-fbd9c80655c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:45:14.893960 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893875 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d32f1343-9c5c-4333-ba98-fbd9c80655c6" (UID: "d32f1343-9c5c-4333-ba98-fbd9c80655c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:45:14.894017 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893970 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d32f1343-9c5c-4333-ba98-fbd9c80655c6" (UID: "d32f1343-9c5c-4333-ba98-fbd9c80655c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:45:14.894017 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.893996 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-data\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:14.894119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.894015 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-workload-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:14.894119 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.894030 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istiod-ca-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:14.894245 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.894097 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d32f1343-9c5c-4333-ba98-fbd9c80655c6" (UID: "d32f1343-9c5c-4333-ba98-fbd9c80655c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:45:14.895902 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.895874 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d32f1343-9c5c-4333-ba98-fbd9c80655c6" (UID: "d32f1343-9c5c-4333-ba98-fbd9c80655c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:45:14.896035 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.895959 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32f1343-9c5c-4333-ba98-fbd9c80655c6-kube-api-access-27g6c" (OuterVolumeSpecName: "kube-api-access-27g6c") pod "d32f1343-9c5c-4333-ba98-fbd9c80655c6" (UID: "d32f1343-9c5c-4333-ba98-fbd9c80655c6"). InnerVolumeSpecName "kube-api-access-27g6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:45:14.896243 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.896217 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "d32f1343-9c5c-4333-ba98-fbd9c80655c6" (UID: "d32f1343-9c5c-4333-ba98-fbd9c80655c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:45:14.896345 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.896293 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d32f1343-9c5c-4333-ba98-fbd9c80655c6" (UID: "d32f1343-9c5c-4333-ba98-fbd9c80655c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:45:14.995032 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.995006 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27g6c\" (UniqueName: \"kubernetes.io/projected/d32f1343-9c5c-4333-ba98-fbd9c80655c6-kube-api-access-27g6c\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:14.995032 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.995032 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-credential-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:14.995219 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.995046 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-podinfo\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:14.995219 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.995057 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-token\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:14.995219 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.995068 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-workload-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:14.995219 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:14.995078 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32f1343-9c5c-4333-ba98-fbd9c80655c6-istio-envoy\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:15.079451 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:15.079418 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:15.708866 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:15.708835 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" Sep 16 01:45:15.709296 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:15.708875 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7" event={"ID":"d32f1343-9c5c-4333-ba98-fbd9c80655c6","Type":"ContainerDied","Data":"313464a891db6d3a91845208a9af49c9cbe7507017eaa4967ff574505e452bd9"} Sep 16 01:45:15.709296 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:15.708934 2558 scope.go:117] "RemoveContainer" containerID="c443adcfc0ba8c7384ce6b19fb5d72abe982c4f2f602bd75730dcc796714ce3a" Sep 16 01:45:15.736213 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:15.736187 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7"] Sep 16 01:45:15.743346 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:15.743317 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f73a36-openshift-default-7568f55688-vfdn7"] Sep 16 01:45:15.773367 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:15.773330 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:16.295611 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:16.295581 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" path="/var/lib/kubelet/pods/d32f1343-9c5c-4333-ba98-fbd9c80655c6/volumes" Sep 16 01:45:16.420030 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:16.419996 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:26.514270 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:26.514240 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:45:27.269057 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:27.269020 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:28.825828 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:28.825796 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:28.855103 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:28.855070 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89"] Sep 16 01:45:28.855407 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:28.855378 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" podUID="32fd6a87-8388-4175-b327-15dee2429085" containerName="payload-pre-processing" containerID="cri-o://799070baefe86528eff4b45a3641e3429f424310d97ac1865928406259137ec0" gracePeriod=30 Sep 16 01:45:28.926035 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:28.926007 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:28.999367 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:28.999340 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:29.049368 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.049341 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:29.128340 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.128316 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" Sep 16 01:45:29.220411 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.220378 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fd6a87-8388-4175-b327-15dee2429085-plugins-config-volume\") pod \"32fd6a87-8388-4175-b327-15dee2429085\" (UID: \"32fd6a87-8388-4175-b327-15dee2429085\") " Sep 16 01:45:29.220617 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.220509 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpzwq\" (UniqueName: \"kubernetes.io/projected/32fd6a87-8388-4175-b327-15dee2429085-kube-api-access-gpzwq\") pod \"32fd6a87-8388-4175-b327-15dee2429085\" (UID: \"32fd6a87-8388-4175-b327-15dee2429085\") " Sep 16 01:45:29.220826 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.220802 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32fd6a87-8388-4175-b327-15dee2429085-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "32fd6a87-8388-4175-b327-15dee2429085" (UID: "32fd6a87-8388-4175-b327-15dee2429085"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:45:29.222767 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.222735 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fd6a87-8388-4175-b327-15dee2429085-kube-api-access-gpzwq" (OuterVolumeSpecName: "kube-api-access-gpzwq") pod "32fd6a87-8388-4175-b327-15dee2429085" (UID: "32fd6a87-8388-4175-b327-15dee2429085"). InnerVolumeSpecName "kube-api-access-gpzwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:45:29.321965 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.321923 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpzwq\" (UniqueName: \"kubernetes.io/projected/32fd6a87-8388-4175-b327-15dee2429085-kube-api-access-gpzwq\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:29.321965 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.321964 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fd6a87-8388-4175-b327-15dee2429085-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:29.390779 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.390755 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:29.798994 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.798912 2558 generic.go:356] "Generic (PLEG): container finished" podID="32fd6a87-8388-4175-b327-15dee2429085" containerID="799070baefe86528eff4b45a3641e3429f424310d97ac1865928406259137ec0" exitCode=0 Sep 16 01:45:29.798994 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.798956 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" Sep 16 01:45:29.798994 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.798964 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" event={"ID":"32fd6a87-8388-4175-b327-15dee2429085","Type":"ContainerDied","Data":"799070baefe86528eff4b45a3641e3429f424310d97ac1865928406259137ec0"} Sep 16 01:45:29.799231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.799024 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89" event={"ID":"32fd6a87-8388-4175-b327-15dee2429085","Type":"ContainerDied","Data":"0b1ad2d5f171140d4d6f04fe15f9f415941a69287a5ad4af79192988ea993e75"} Sep 16 01:45:29.799231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.799052 2558 scope.go:117] "RemoveContainer" containerID="799070baefe86528eff4b45a3641e3429f424310d97ac1865928406259137ec0" Sep 16 01:45:29.809712 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.809696 2558 scope.go:117] "RemoveContainer" containerID="799070baefe86528eff4b45a3641e3429f424310d97ac1865928406259137ec0" Sep 16 01:45:29.809981 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:45:29.809962 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799070baefe86528eff4b45a3641e3429f424310d97ac1865928406259137ec0\": container with ID starting with 799070baefe86528eff4b45a3641e3429f424310d97ac1865928406259137ec0 not found: ID does not exist" containerID="799070baefe86528eff4b45a3641e3429f424310d97ac1865928406259137ec0" Sep 16 01:45:29.810060 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.809990 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799070baefe86528eff4b45a3641e3429f424310d97ac1865928406259137ec0"} err="failed to get container status \"799070baefe86528eff4b45a3641e3429f424310d97ac1865928406259137ec0\": rpc error: code = NotFound desc = could not find container \"799070baefe86528eff4b45a3641e3429f424310d97ac1865928406259137ec0\": container with ID starting with 799070baefe86528eff4b45a3641e3429f424310d97ac1865928406259137ec0 not found: ID does not exist" Sep 16 01:45:29.823383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.823358 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89"] Sep 16 01:45:29.825874 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.825854 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d230b-c9484646-xhd89"] Sep 16 01:45:29.960630 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:29.960601 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:30.295981 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:30.295947 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32fd6a87-8388-4175-b327-15dee2429085" path="/var/lib/kubelet/pods/32fd6a87-8388-4175-b327-15dee2429085/volumes" Sep 16 01:45:30.669578 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:30.669548 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:34.387261 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:34.387226 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64"] Sep 16 01:45:34.480448 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:34.480417 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:45:38.119076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:38.119044 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:45:49.407329 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.407269 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" podUID="5038efde-3484-4cda-80c3-482e794daf28" containerName="main" containerID="cri-o://5cb713913d6de53d2571dd7fb415da8ab2b437e288ef4c1f3261a604be8cdc20" gracePeriod=45 Sep 16 01:45:49.666989 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.666932 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:45:49.804440 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.804394 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zhlc\" (UniqueName: \"kubernetes.io/projected/5038efde-3484-4cda-80c3-482e794daf28-kube-api-access-9zhlc\") pod \"5038efde-3484-4cda-80c3-482e794daf28\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " Sep 16 01:45:49.804626 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.804470 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-model-cache\") pod \"5038efde-3484-4cda-80c3-482e794daf28\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " Sep 16 01:45:49.804626 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.804513 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5038efde-3484-4cda-80c3-482e794daf28-tls-certs\") pod \"5038efde-3484-4cda-80c3-482e794daf28\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " Sep 16 01:45:49.804626 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.804535 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-dshm\") pod \"5038efde-3484-4cda-80c3-482e794daf28\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " Sep 16 01:45:49.804626 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.804569 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-tmp-dir\") pod \"5038efde-3484-4cda-80c3-482e794daf28\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " Sep 16 01:45:49.804626 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.804589 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-home\") pod \"5038efde-3484-4cda-80c3-482e794daf28\" (UID: \"5038efde-3484-4cda-80c3-482e794daf28\") " Sep 16 01:45:49.804891 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.804777 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-model-cache" (OuterVolumeSpecName: "model-cache") pod "5038efde-3484-4cda-80c3-482e794daf28" (UID: "5038efde-3484-4cda-80c3-482e794daf28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:45:49.804958 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.804922 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-home" (OuterVolumeSpecName: "home") pod "5038efde-3484-4cda-80c3-482e794daf28" (UID: "5038efde-3484-4cda-80c3-482e794daf28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:45:49.804958 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.804917 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5038efde-3484-4cda-80c3-482e794daf28" (UID: "5038efde-3484-4cda-80c3-482e794daf28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:45:49.805059 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.804990 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-model-cache\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:49.806903 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.806874 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-dshm" (OuterVolumeSpecName: "dshm") pod "5038efde-3484-4cda-80c3-482e794daf28" (UID: "5038efde-3484-4cda-80c3-482e794daf28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:45:49.807124 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.807101 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5038efde-3484-4cda-80c3-482e794daf28-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5038efde-3484-4cda-80c3-482e794daf28" (UID: "5038efde-3484-4cda-80c3-482e794daf28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:45:49.807235 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.807128 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5038efde-3484-4cda-80c3-482e794daf28-kube-api-access-9zhlc" (OuterVolumeSpecName: "kube-api-access-9zhlc") pod "5038efde-3484-4cda-80c3-482e794daf28" (UID: "5038efde-3484-4cda-80c3-482e794daf28"). InnerVolumeSpecName "kube-api-access-9zhlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:45:49.906230 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.906208 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zhlc\" (UniqueName: \"kubernetes.io/projected/5038efde-3484-4cda-80c3-482e794daf28-kube-api-access-9zhlc\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:49.906230 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.906228 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5038efde-3484-4cda-80c3-482e794daf28-tls-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:49.906368 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.906239 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-dshm\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:49.906368 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.906247 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-tmp-dir\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:49.906368 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.906254 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5038efde-3484-4cda-80c3-482e794daf28-home\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:45:49.924495 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.924436 2558 generic.go:356] "Generic (PLEG): container finished" podID="5038efde-3484-4cda-80c3-482e794daf28" containerID="5cb713913d6de53d2571dd7fb415da8ab2b437e288ef4c1f3261a604be8cdc20" exitCode=0 Sep 16 01:45:49.924495 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.924480 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" Sep 16 01:45:49.924638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.924524 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" event={"ID":"5038efde-3484-4cda-80c3-482e794daf28","Type":"ContainerDied","Data":"5cb713913d6de53d2571dd7fb415da8ab2b437e288ef4c1f3261a604be8cdc20"} Sep 16 01:45:49.924638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.924569 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64" event={"ID":"5038efde-3484-4cda-80c3-482e794daf28","Type":"ContainerDied","Data":"acffa11160f375832dc9d0d3ee426d07a9320c4cdfbfd202804fa8fd010c87ae"} Sep 16 01:45:49.924638 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.924591 2558 scope.go:117] "RemoveContainer" containerID="5cb713913d6de53d2571dd7fb415da8ab2b437e288ef4c1f3261a604be8cdc20" Sep 16 01:45:49.933961 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.933940 2558 scope.go:117] "RemoveContainer" containerID="5cb713913d6de53d2571dd7fb415da8ab2b437e288ef4c1f3261a604be8cdc20" Sep 16 01:45:49.934305 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:45:49.934287 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb713913d6de53d2571dd7fb415da8ab2b437e288ef4c1f3261a604be8cdc20\": container with ID starting with 5cb713913d6de53d2571dd7fb415da8ab2b437e288ef4c1f3261a604be8cdc20 not found: ID does not exist" containerID="5cb713913d6de53d2571dd7fb415da8ab2b437e288ef4c1f3261a604be8cdc20" Sep 16 01:45:49.934367 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.934312 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb713913d6de53d2571dd7fb415da8ab2b437e288ef4c1f3261a604be8cdc20"} err="failed to get container status \"5cb713913d6de53d2571dd7fb415da8ab2b437e288ef4c1f3261a604be8cdc20\": rpc error: code = NotFound desc = could not find container \"5cb713913d6de53d2571dd7fb415da8ab2b437e288ef4c1f3261a604be8cdc20\": container with ID starting with 5cb713913d6de53d2571dd7fb415da8ab2b437e288ef4c1f3261a604be8cdc20 not found: ID does not exist" Sep 16 01:45:49.946038 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.946012 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64"] Sep 16 01:45:49.949125 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:49.949092 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-8d230b/ipp-route-8d230b-kserve-864f44bc87-jlj64"] Sep 16 01:45:50.296501 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:45:50.296424 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5038efde-3484-4cda-80c3-482e794daf28" path="/var/lib/kubelet/pods/5038efde-3484-4cda-80c3-482e794daf28/volumes" Sep 16 01:46:10.381893 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:10.381857 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:46:10.388804 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:10.388778 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:46:10.620094 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:10.620060 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:11.453447 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:11.453412 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:16.868674 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:16.868643 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:16.967669 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:16.967635 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:17.165005 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:17.164927 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:18.432259 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:18.432228 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:19.170573 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:19.170543 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:19.766948 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:19.766918 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:28.069607 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.069572 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:28.093885 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.093856 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5"] Sep 16 01:46:28.094141 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.094090 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" podUID="ab5a3df4-818f-4c84-8187-c29a4ca5b7e3" containerName="payload-processing" containerID="cri-o://19f8cce45e7f1e4df908ebb7de7614b11e0443701c01c4563e7525a4db8eb161" gracePeriod=30 Sep 16 01:46:28.168941 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.168887 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:28.243188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.243124 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:28.302605 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.302572 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:28.357060 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.357030 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" Sep 16 01:46:28.444310 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.444273 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5a3df4-818f-4c84-8187-c29a4ca5b7e3-plugins-config-volume\") pod \"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3\" (UID: \"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3\") " Sep 16 01:46:28.444492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.444420 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmkpz\" (UniqueName: \"kubernetes.io/projected/ab5a3df4-818f-4c84-8187-c29a4ca5b7e3-kube-api-access-gmkpz\") pod \"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3\" (UID: \"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3\") " Sep 16 01:46:28.444985 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.444942 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5a3df4-818f-4c84-8187-c29a4ca5b7e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab5a3df4-818f-4c84-8187-c29a4ca5b7e3" (UID: "ab5a3df4-818f-4c84-8187-c29a4ca5b7e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:46:28.447051 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.447018 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5a3df4-818f-4c84-8187-c29a4ca5b7e3-kube-api-access-gmkpz" (OuterVolumeSpecName: "kube-api-access-gmkpz") pod "ab5a3df4-818f-4c84-8187-c29a4ca5b7e3" (UID: "ab5a3df4-818f-4c84-8187-c29a4ca5b7e3"). InnerVolumeSpecName "kube-api-access-gmkpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:46:28.545623 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.545588 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5a3df4-818f-4c84-8187-c29a4ca5b7e3-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:28.545623 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.545616 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmkpz\" (UniqueName: \"kubernetes.io/projected/ab5a3df4-818f-4c84-8187-c29a4ca5b7e3-kube-api-access-gmkpz\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:28.643864 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.643833 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:28.772781 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:28.772743 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:29.162006 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:29.161971 2558 generic.go:356] "Generic (PLEG): container finished" podID="ab5a3df4-818f-4c84-8187-c29a4ca5b7e3" containerID="19f8cce45e7f1e4df908ebb7de7614b11e0443701c01c4563e7525a4db8eb161" exitCode=0 Sep 16 01:46:29.162444 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:29.162016 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" Sep 16 01:46:29.162444 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:29.162057 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" event={"ID":"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3","Type":"ContainerDied","Data":"19f8cce45e7f1e4df908ebb7de7614b11e0443701c01c4563e7525a4db8eb161"} Sep 16 01:46:29.162444 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:29.162103 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5" event={"ID":"ab5a3df4-818f-4c84-8187-c29a4ca5b7e3","Type":"ContainerDied","Data":"571829d2abc44d8be8888c337c2fb25e6c0863f4adf676f0eed8dd0d07c390c8"} Sep 16 01:46:29.162444 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:29.162124 2558 scope.go:117] "RemoveContainer" containerID="19f8cce45e7f1e4df908ebb7de7614b11e0443701c01c4563e7525a4db8eb161" Sep 16 01:46:29.174661 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:29.174639 2558 scope.go:117] "RemoveContainer" containerID="19f8cce45e7f1e4df908ebb7de7614b11e0443701c01c4563e7525a4db8eb161" Sep 16 01:46:29.175052 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:46:29.175032 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f8cce45e7f1e4df908ebb7de7614b11e0443701c01c4563e7525a4db8eb161\": container with ID starting with 19f8cce45e7f1e4df908ebb7de7614b11e0443701c01c4563e7525a4db8eb161 not found: ID does not exist" containerID="19f8cce45e7f1e4df908ebb7de7614b11e0443701c01c4563e7525a4db8eb161" Sep 16 01:46:29.175120 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:29.175062 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f8cce45e7f1e4df908ebb7de7614b11e0443701c01c4563e7525a4db8eb161"} err="failed to get container status \"19f8cce45e7f1e4df908ebb7de7614b11e0443701c01c4563e7525a4db8eb161\": rpc error: code = NotFound desc = could not find container \"19f8cce45e7f1e4df908ebb7de7614b11e0443701c01c4563e7525a4db8eb161\": container with ID starting with 19f8cce45e7f1e4df908ebb7de7614b11e0443701c01c4563e7525a4db8eb161 not found: ID does not exist" Sep 16 01:46:29.183097 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:29.183074 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5"] Sep 16 01:46:29.187461 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:29.187436 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a42101-5648d454c7-ctpr5"] Sep 16 01:46:29.445372 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:29.445299 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:30.295403 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:30.295371 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab5a3df4-818f-4c84-8187-c29a4ca5b7e3" path="/var/lib/kubelet/pods/ab5a3df4-818f-4c84-8187-c29a4ca5b7e3/volumes" Sep 16 01:46:31.218215 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:31.218183 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:46:34.272675 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:34.272642 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:34.373998 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:34.373944 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:34.525697 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:34.525621 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8"] Sep 16 01:46:34.525860 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:34.525839 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerName="istio-proxy" containerID="cri-o://708aa54ce25df0739d9d504cc018abfee76e09c010db1c0cef8078309fbac8e0" gracePeriod=30 Sep 16 01:46:34.540585 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:34.540552 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:35.062218 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:35.062183 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:35.929683 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:35.929643 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:36.658922 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:36.658887 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:39.613541 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.613511 2558 patch_prober.go:28] interesting pod/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 16 01:46:39.613866 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.613570 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 16 01:46:39.775693 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.775665 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:46:39.847091 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847018 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-data\") pod \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " Sep 16 01:46:39.847091 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847057 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8b29\" (UniqueName: \"kubernetes.io/projected/8c44983e-f5c6-4c73-b8a1-844e47b3390e-kube-api-access-p8b29\") pod \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " Sep 16 01:46:39.847313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847102 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-workload-socket\") pod \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " Sep 16 01:46:39.847313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847190 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-envoy\") pod \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " Sep 16 01:46:39.847313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847205 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-token\") pod \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " Sep 16 01:46:39.847313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847230 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-podinfo\") pod \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " Sep 16 01:46:39.847313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847247 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-workload-certs\") pod \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " Sep 16 01:46:39.847313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847265 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istiod-ca-cert\") pod \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " Sep 16 01:46:39.847313 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847313 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-credential-socket\") pod \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\" (UID: \"8c44983e-f5c6-4c73-b8a1-844e47b3390e\") " Sep 16 01:46:39.847681 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847314 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-data" (OuterVolumeSpecName: "istio-data") pod "8c44983e-f5c6-4c73-b8a1-844e47b3390e" (UID: "8c44983e-f5c6-4c73-b8a1-844e47b3390e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:46:39.847681 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847539 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8c44983e-f5c6-4c73-b8a1-844e47b3390e" (UID: "8c44983e-f5c6-4c73-b8a1-844e47b3390e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:46:39.847681 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847595 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-data\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:39.847681 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847628 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8c44983e-f5c6-4c73-b8a1-844e47b3390e" (UID: "8c44983e-f5c6-4c73-b8a1-844e47b3390e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:46:39.847896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847773 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8c44983e-f5c6-4c73-b8a1-844e47b3390e" (UID: "8c44983e-f5c6-4c73-b8a1-844e47b3390e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:46:39.847896 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.847851 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8c44983e-f5c6-4c73-b8a1-844e47b3390e" (UID: "8c44983e-f5c6-4c73-b8a1-844e47b3390e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:46:39.849522 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.849495 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8c44983e-f5c6-4c73-b8a1-844e47b3390e" (UID: "8c44983e-f5c6-4c73-b8a1-844e47b3390e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:46:39.849636 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.849516 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8c44983e-f5c6-4c73-b8a1-844e47b3390e" (UID: "8c44983e-f5c6-4c73-b8a1-844e47b3390e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:46:39.849884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.849861 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-token" (OuterVolumeSpecName: "istio-token") pod "8c44983e-f5c6-4c73-b8a1-844e47b3390e" (UID: "8c44983e-f5c6-4c73-b8a1-844e47b3390e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:46:39.849884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.849871 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c44983e-f5c6-4c73-b8a1-844e47b3390e-kube-api-access-p8b29" (OuterVolumeSpecName: "kube-api-access-p8b29") pod "8c44983e-f5c6-4c73-b8a1-844e47b3390e" (UID: "8c44983e-f5c6-4c73-b8a1-844e47b3390e"). InnerVolumeSpecName "kube-api-access-p8b29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:46:39.948774 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.948745 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-workload-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:39.948774 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.948772 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istiod-ca-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:39.948928 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.948782 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-credential-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:39.948928 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.948791 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8b29\" (UniqueName: \"kubernetes.io/projected/8c44983e-f5c6-4c73-b8a1-844e47b3390e-kube-api-access-p8b29\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:39.948928 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.948800 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-workload-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:39.948928 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.948810 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-envoy\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:39.948928 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.948818 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-token\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:39.948928 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:39.948825 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c44983e-f5c6-4c73-b8a1-844e47b3390e-istio-podinfo\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:40.229388 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:40.229355 2558 generic.go:356] "Generic (PLEG): container finished" podID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerID="708aa54ce25df0739d9d504cc018abfee76e09c010db1c0cef8078309fbac8e0" exitCode=0 Sep 16 01:46:40.229536 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:40.229411 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" Sep 16 01:46:40.229536 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:40.229451 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" event={"ID":"8c44983e-f5c6-4c73-b8a1-844e47b3390e","Type":"ContainerDied","Data":"708aa54ce25df0739d9d504cc018abfee76e09c010db1c0cef8078309fbac8e0"} Sep 16 01:46:40.229536 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:40.229531 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8" event={"ID":"8c44983e-f5c6-4c73-b8a1-844e47b3390e","Type":"ContainerDied","Data":"ce58f231d5bb634141397390721a0082c59c3cc5550e257044bc2ff85e19c683"} Sep 16 01:46:40.229689 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:40.229554 2558 scope.go:117] "RemoveContainer" containerID="708aa54ce25df0739d9d504cc018abfee76e09c010db1c0cef8078309fbac8e0" Sep 16 01:46:40.239945 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:40.239926 2558 scope.go:117] "RemoveContainer" containerID="9eb6bd558cc77bcab2a0364781a3a66f85ed1f2bbb6fa9cb067766657539f460" Sep 16 01:46:40.256149 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:40.249978 2558 scope.go:117] "RemoveContainer" containerID="708aa54ce25df0739d9d504cc018abfee76e09c010db1c0cef8078309fbac8e0" Sep 16 01:46:40.256149 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:40.252115 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8"] Sep 16 01:46:40.256149 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:40.256074 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a42101-openshift-default-64975f9d58-m2ps8"] Sep 16 01:46:40.260085 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:46:40.260064 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708aa54ce25df0739d9d504cc018abfee76e09c010db1c0cef8078309fbac8e0\": container with ID starting with 708aa54ce25df0739d9d504cc018abfee76e09c010db1c0cef8078309fbac8e0 not found: ID does not exist" containerID="708aa54ce25df0739d9d504cc018abfee76e09c010db1c0cef8078309fbac8e0" Sep 16 01:46:40.260207 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:40.260092 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708aa54ce25df0739d9d504cc018abfee76e09c010db1c0cef8078309fbac8e0"} err="failed to get container status \"708aa54ce25df0739d9d504cc018abfee76e09c010db1c0cef8078309fbac8e0\": rpc error: code = NotFound desc = could not find container \"708aa54ce25df0739d9d504cc018abfee76e09c010db1c0cef8078309fbac8e0\": container with ID starting with 708aa54ce25df0739d9d504cc018abfee76e09c010db1c0cef8078309fbac8e0 not found: ID does not exist" Sep 16 01:46:40.260207 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:40.260115 2558 scope.go:117] "RemoveContainer" containerID="9eb6bd558cc77bcab2a0364781a3a66f85ed1f2bbb6fa9cb067766657539f460" Sep 16 01:46:40.260410 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:46:40.260391 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb6bd558cc77bcab2a0364781a3a66f85ed1f2bbb6fa9cb067766657539f460\": container with ID starting with 9eb6bd558cc77bcab2a0364781a3a66f85ed1f2bbb6fa9cb067766657539f460 not found: ID does not exist" containerID="9eb6bd558cc77bcab2a0364781a3a66f85ed1f2bbb6fa9cb067766657539f460" Sep 16 01:46:40.260468 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:40.260414 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb6bd558cc77bcab2a0364781a3a66f85ed1f2bbb6fa9cb067766657539f460"} err="failed to get container status \"9eb6bd558cc77bcab2a0364781a3a66f85ed1f2bbb6fa9cb067766657539f460\": rpc error: code = NotFound desc = could not find container \"9eb6bd558cc77bcab2a0364781a3a66f85ed1f2bbb6fa9cb067766657539f460\": container with ID starting with 9eb6bd558cc77bcab2a0364781a3a66f85ed1f2bbb6fa9cb067766657539f460 not found: ID does not exist" Sep 16 01:46:40.295132 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:40.295108 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" path="/var/lib/kubelet/pods/8c44983e-f5c6-4c73-b8a1-844e47b3390e/volumes" Sep 16 01:46:45.340227 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.340187 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:45.341528 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.341500 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn"] Sep 16 01:46:45.343850 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.343827 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cbb499-29825385-4qfzn"] Sep 16 01:46:45.370812 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.370778 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7"] Sep 16 01:46:45.371046 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.371019 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" podUID="8e572b8a-1c4c-497f-8b11-195b92f33c0c" containerName="payload-processing" containerID="cri-o://4ec6e73fe87ac8f74a78118858977d04246b8e77fd060d5030138dcd582d0bf0" gracePeriod=30 Sep 16 01:46:45.376357 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.376334 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z"] Sep 16 01:46:45.376856 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.376827 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" podUID="5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8" containerName="payload-pre-processing" containerID="cri-o://9494fc825f17f4b8ca8043e04604ed375045f57f5e6d8dfde5f6f26f5c61f4a2" gracePeriod=30 Sep 16 01:46:45.398417 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.398391 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:45.457679 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.457653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:45.515867 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.515805 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:45.569535 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.569490 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:45.648197 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.648152 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" Sep 16 01:46:45.675643 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.675616 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" Sep 16 01:46:45.799600 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.799567 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfwl8\" (UniqueName: \"kubernetes.io/projected/5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8-kube-api-access-cfwl8\") pod \"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8\" (UID: \"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8\") " Sep 16 01:46:45.799764 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.799654 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e572b8a-1c4c-497f-8b11-195b92f33c0c-plugins-config-volume\") pod \"8e572b8a-1c4c-497f-8b11-195b92f33c0c\" (UID: \"8e572b8a-1c4c-497f-8b11-195b92f33c0c\") " Sep 16 01:46:45.799764 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.799684 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8-plugins-config-volume\") pod \"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8\" (UID: \"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8\") " Sep 16 01:46:45.799764 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.799731 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcz4r\" (UniqueName: \"kubernetes.io/projected/8e572b8a-1c4c-497f-8b11-195b92f33c0c-kube-api-access-zcz4r\") pod \"8e572b8a-1c4c-497f-8b11-195b92f33c0c\" (UID: \"8e572b8a-1c4c-497f-8b11-195b92f33c0c\") " Sep 16 01:46:45.800189 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.800131 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e572b8a-1c4c-497f-8b11-195b92f33c0c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e572b8a-1c4c-497f-8b11-195b92f33c0c" (UID: "8e572b8a-1c4c-497f-8b11-195b92f33c0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:46:45.800282 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.800138 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8" (UID: "5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:46:45.801830 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.801808 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8-kube-api-access-cfwl8" (OuterVolumeSpecName: "kube-api-access-cfwl8") pod "5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8" (UID: "5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8"). InnerVolumeSpecName "kube-api-access-cfwl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:46:45.801983 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.801965 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e572b8a-1c4c-497f-8b11-195b92f33c0c-kube-api-access-zcz4r" (OuterVolumeSpecName: "kube-api-access-zcz4r") pod "8e572b8a-1c4c-497f-8b11-195b92f33c0c" (UID: "8e572b8a-1c4c-497f-8b11-195b92f33c0c"). InnerVolumeSpecName "kube-api-access-zcz4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:46:45.900452 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.900425 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfwl8\" (UniqueName: \"kubernetes.io/projected/5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8-kube-api-access-cfwl8\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:45.900452 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.900451 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e572b8a-1c4c-497f-8b11-195b92f33c0c-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:45.900621 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.900460 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:45.900621 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.900468 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcz4r\" (UniqueName: \"kubernetes.io/projected/8e572b8a-1c4c-497f-8b11-195b92f33c0c-kube-api-access-zcz4r\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:45.955230 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:45.955205 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:46.165031 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.164950 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:46.267802 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.267770 2558 generic.go:356] "Generic (PLEG): container finished" podID="5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8" containerID="9494fc825f17f4b8ca8043e04604ed375045f57f5e6d8dfde5f6f26f5c61f4a2" exitCode=0 Sep 16 01:46:46.267975 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.267831 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" Sep 16 01:46:46.267975 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.267869 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" event={"ID":"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8","Type":"ContainerDied","Data":"9494fc825f17f4b8ca8043e04604ed375045f57f5e6d8dfde5f6f26f5c61f4a2"} Sep 16 01:46:46.267975 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.267964 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z" event={"ID":"5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8","Type":"ContainerDied","Data":"96e3a3838f560c273a341a9ded3a3125e0dcd6e5b6b21db997f238fdd109bb3f"} Sep 16 01:46:46.268132 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.267997 2558 scope.go:117] "RemoveContainer" containerID="9494fc825f17f4b8ca8043e04604ed375045f57f5e6d8dfde5f6f26f5c61f4a2" Sep 16 01:46:46.269519 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.269488 2558 generic.go:356] "Generic (PLEG): container finished" podID="8e572b8a-1c4c-497f-8b11-195b92f33c0c" containerID="4ec6e73fe87ac8f74a78118858977d04246b8e77fd060d5030138dcd582d0bf0" exitCode=0 Sep 16 01:46:46.269617 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.269546 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" event={"ID":"8e572b8a-1c4c-497f-8b11-195b92f33c0c","Type":"ContainerDied","Data":"4ec6e73fe87ac8f74a78118858977d04246b8e77fd060d5030138dcd582d0bf0"} Sep 16 01:46:46.269617 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.269582 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" event={"ID":"8e572b8a-1c4c-497f-8b11-195b92f33c0c","Type":"ContainerDied","Data":"1d4086aeaf1d4bceab3086b41cac6114296638c5a361d5d15f4fc1fd47185272"} Sep 16 01:46:46.269617 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.269549 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7" Sep 16 01:46:46.279965 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.279947 2558 scope.go:117] "RemoveContainer" containerID="9494fc825f17f4b8ca8043e04604ed375045f57f5e6d8dfde5f6f26f5c61f4a2" Sep 16 01:46:46.280292 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:46:46.280270 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9494fc825f17f4b8ca8043e04604ed375045f57f5e6d8dfde5f6f26f5c61f4a2\": container with ID starting with 9494fc825f17f4b8ca8043e04604ed375045f57f5e6d8dfde5f6f26f5c61f4a2 not found: ID does not exist" containerID="9494fc825f17f4b8ca8043e04604ed375045f57f5e6d8dfde5f6f26f5c61f4a2" Sep 16 01:46:46.280371 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.280300 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9494fc825f17f4b8ca8043e04604ed375045f57f5e6d8dfde5f6f26f5c61f4a2"} err="failed to get container status \"9494fc825f17f4b8ca8043e04604ed375045f57f5e6d8dfde5f6f26f5c61f4a2\": rpc error: code = NotFound desc = could not find container \"9494fc825f17f4b8ca8043e04604ed375045f57f5e6d8dfde5f6f26f5c61f4a2\": container with ID starting with 9494fc825f17f4b8ca8043e04604ed375045f57f5e6d8dfde5f6f26f5c61f4a2 not found: ID does not exist" Sep 16 01:46:46.280371 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.280323 2558 scope.go:117] "RemoveContainer" containerID="4ec6e73fe87ac8f74a78118858977d04246b8e77fd060d5030138dcd582d0bf0" Sep 16 01:46:46.289143 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.289128 2558 scope.go:117] "RemoveContainer" containerID="4ec6e73fe87ac8f74a78118858977d04246b8e77fd060d5030138dcd582d0bf0" Sep 16 01:46:46.289429 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:46:46.289409 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec6e73fe87ac8f74a78118858977d04246b8e77fd060d5030138dcd582d0bf0\": container with ID starting with 4ec6e73fe87ac8f74a78118858977d04246b8e77fd060d5030138dcd582d0bf0 not found: ID does not exist" containerID="4ec6e73fe87ac8f74a78118858977d04246b8e77fd060d5030138dcd582d0bf0" Sep 16 01:46:46.289485 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.289436 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec6e73fe87ac8f74a78118858977d04246b8e77fd060d5030138dcd582d0bf0"} err="failed to get container status \"4ec6e73fe87ac8f74a78118858977d04246b8e77fd060d5030138dcd582d0bf0\": rpc error: code = NotFound desc = could not find container \"4ec6e73fe87ac8f74a78118858977d04246b8e77fd060d5030138dcd582d0bf0\": container with ID starting with 4ec6e73fe87ac8f74a78118858977d04246b8e77fd060d5030138dcd582d0bf0 not found: ID does not exist" Sep 16 01:46:46.297347 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.297319 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c553b16-a983-46b2-ba04-a19cccebc22f" path="/var/lib/kubelet/pods/6c553b16-a983-46b2-ba04-a19cccebc22f/volumes" Sep 16 01:46:46.297560 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.297546 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z"] Sep 16 01:46:46.299446 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.299427 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cbb499-6dc9d464c4-t8b5z"] Sep 16 01:46:46.307993 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.307974 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7"] Sep 16 01:46:46.310207 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:46.310186 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cbb499-66f5446547-fx2z7"] Sep 16 01:46:48.294439 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:48.294410 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8" path="/var/lib/kubelet/pods/5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8/volumes" Sep 16 01:46:48.294809 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:48.294759 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e572b8a-1c4c-497f-8b11-195b92f33c0c" path="/var/lib/kubelet/pods/8e572b8a-1c4c-497f-8b11-195b92f33c0c/volumes" Sep 16 01:46:48.522046 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:48.522015 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:46:51.869845 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:51.869806 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:52.121726 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:52.121638 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr"] Sep 16 01:46:52.121891 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:52.121864 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" podUID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerName="istio-proxy" containerID="cri-o://bbd68e82209079453143de631e622817eadec545f17a3e8f92aef8abffbc01a8" gracePeriod=30 Sep 16 01:46:53.277980 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:53.277933 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:46:57.344225 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.344188 2558 generic.go:356] "Generic (PLEG): container finished" podID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerID="bbd68e82209079453143de631e622817eadec545f17a3e8f92aef8abffbc01a8" exitCode=0 Sep 16 01:46:57.344599 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.344250 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" event={"ID":"bbc26edd-b9e5-41b5-a015-e2f9f63e2586","Type":"ContainerDied","Data":"bbd68e82209079453143de631e622817eadec545f17a3e8f92aef8abffbc01a8"} Sep 16 01:46:57.344599 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.344315 2558 scope.go:117] "RemoveContainer" containerID="265c4193c1e521622e3ad93ff62029c4b8f8f6cd81de00980804cbbbf0c9c110" Sep 16 01:46:57.379067 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.379044 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:46:57.500073 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.499994 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-credential-socket\") pod \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " Sep 16 01:46:57.500073 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.500026 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-workload-socket\") pod \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " Sep 16 01:46:57.500334 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.500089 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-envoy\") pod \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " Sep 16 01:46:57.500334 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.500123 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-748l7\" (UniqueName: \"kubernetes.io/projected/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-kube-api-access-748l7\") pod \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " Sep 16 01:46:57.500334 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.500176 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-workload-certs\") pod \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " Sep 16 01:46:57.500334 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.500200 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-data\") pod \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " Sep 16 01:46:57.500334 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.500229 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-podinfo\") pod \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " Sep 16 01:46:57.500334 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.500254 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istiod-ca-cert\") pod \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " Sep 16 01:46:57.500334 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.500271 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-token\") pod \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\" (UID: \"bbc26edd-b9e5-41b5-a015-e2f9f63e2586\") " Sep 16 01:46:57.500682 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.500384 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bbc26edd-b9e5-41b5-a015-e2f9f63e2586" (UID: "bbc26edd-b9e5-41b5-a015-e2f9f63e2586"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:46:57.500682 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.500468 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bbc26edd-b9e5-41b5-a015-e2f9f63e2586" (UID: "bbc26edd-b9e5-41b5-a015-e2f9f63e2586"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:46:57.500682 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.500534 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-credential-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:57.500682 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.500669 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-data" (OuterVolumeSpecName: "istio-data") pod "bbc26edd-b9e5-41b5-a015-e2f9f63e2586" (UID: "bbc26edd-b9e5-41b5-a015-e2f9f63e2586"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:46:57.500909 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.500860 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bbc26edd-b9e5-41b5-a015-e2f9f63e2586" (UID: "bbc26edd-b9e5-41b5-a015-e2f9f63e2586"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:46:57.501087 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.501061 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bbc26edd-b9e5-41b5-a015-e2f9f63e2586" (UID: "bbc26edd-b9e5-41b5-a015-e2f9f63e2586"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:46:57.502763 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.502724 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bbc26edd-b9e5-41b5-a015-e2f9f63e2586" (UID: "bbc26edd-b9e5-41b5-a015-e2f9f63e2586"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:46:57.502846 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.502808 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-kube-api-access-748l7" (OuterVolumeSpecName: "kube-api-access-748l7") pod "bbc26edd-b9e5-41b5-a015-e2f9f63e2586" (UID: "bbc26edd-b9e5-41b5-a015-e2f9f63e2586"). InnerVolumeSpecName "kube-api-access-748l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:46:57.502991 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.502973 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-token" (OuterVolumeSpecName: "istio-token") pod "bbc26edd-b9e5-41b5-a015-e2f9f63e2586" (UID: "bbc26edd-b9e5-41b5-a015-e2f9f63e2586"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:46:57.503029 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.502984 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bbc26edd-b9e5-41b5-a015-e2f9f63e2586" (UID: "bbc26edd-b9e5-41b5-a015-e2f9f63e2586"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:46:57.601491 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.601460 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istiod-ca-cert\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:57.601491 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.601484 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-token\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:57.601639 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.601497 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-workload-socket\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:57.601639 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.601508 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-envoy\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:57.601639 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.601521 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-748l7\" (UniqueName: \"kubernetes.io/projected/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-kube-api-access-748l7\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:57.601639 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.601532 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-workload-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:57.601639 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.601544 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-data\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:57.601639 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:57.601555 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc26edd-b9e5-41b5-a015-e2f9f63e2586-istio-podinfo\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:46:58.350994 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:58.350968 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" Sep 16 01:46:58.351379 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:58.351001 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr" event={"ID":"bbc26edd-b9e5-41b5-a015-e2f9f63e2586","Type":"ContainerDied","Data":"094e6f0ee9a0439284fb346ae4b984f7240720bad37aae960430f4f7fb6b66e8"} Sep 16 01:46:58.351379 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:58.351059 2558 scope.go:117] "RemoveContainer" containerID="bbd68e82209079453143de631e622817eadec545f17a3e8f92aef8abffbc01a8" Sep 16 01:46:58.367266 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:58.367241 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr"] Sep 16 01:46:58.373173 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:46:58.373130 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cbb499-openshift-default-74d67c6744-dbfrr"] Sep 16 01:47:00.295101 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:00.295058 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" path="/var/lib/kubelet/pods/bbc26edd-b9e5-41b5-a015-e2f9f63e2586/volumes" Sep 16 01:47:03.080118 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.080077 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp"] Sep 16 01:47:03.085500 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.085468 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0ec73-29825385-xhlbp"] Sep 16 01:47:03.087940 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.087886 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:47:03.100625 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.100591 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd"] Sep 16 01:47:03.100942 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.100883 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" podUID="9774ea02-8641-4166-9c73-4551543fe5f3" containerName="payload-processing" containerID="cri-o://c67a17a25be544d77a71bb2eb4e950f1e22793ec26be3689b50bbc58109f8987" gracePeriod=30 Sep 16 01:47:03.192089 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.192056 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:47:03.314901 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.314870 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:47:03.363482 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.363457 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" Sep 16 01:47:03.383134 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.382830 2558 generic.go:356] "Generic (PLEG): container finished" podID="9774ea02-8641-4166-9c73-4551543fe5f3" containerID="c67a17a25be544d77a71bb2eb4e950f1e22793ec26be3689b50bbc58109f8987" exitCode=0 Sep 16 01:47:03.383134 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.382937 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" Sep 16 01:47:03.383134 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.382937 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" event={"ID":"9774ea02-8641-4166-9c73-4551543fe5f3","Type":"ContainerDied","Data":"c67a17a25be544d77a71bb2eb4e950f1e22793ec26be3689b50bbc58109f8987"} Sep 16 01:47:03.383134 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.382987 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd" event={"ID":"9774ea02-8641-4166-9c73-4551543fe5f3","Type":"ContainerDied","Data":"c91fd7bb71284b70a9279214b06495812874ae445ff6e0cc657f248677b04680"} Sep 16 01:47:03.383134 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.383019 2558 scope.go:117] "RemoveContainer" containerID="c67a17a25be544d77a71bb2eb4e950f1e22793ec26be3689b50bbc58109f8987" Sep 16 01:47:03.388123 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.388096 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:47:03.394231 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.394214 2558 scope.go:117] "RemoveContainer" containerID="c67a17a25be544d77a71bb2eb4e950f1e22793ec26be3689b50bbc58109f8987" Sep 16 01:47:03.394514 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:47:03.394488 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67a17a25be544d77a71bb2eb4e950f1e22793ec26be3689b50bbc58109f8987\": container with ID starting with c67a17a25be544d77a71bb2eb4e950f1e22793ec26be3689b50bbc58109f8987 not found: ID does not exist" containerID="c67a17a25be544d77a71bb2eb4e950f1e22793ec26be3689b50bbc58109f8987" Sep 16 01:47:03.394567 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.394526 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67a17a25be544d77a71bb2eb4e950f1e22793ec26be3689b50bbc58109f8987"} err="failed to get container status \"c67a17a25be544d77a71bb2eb4e950f1e22793ec26be3689b50bbc58109f8987\": rpc error: code = NotFound desc = could not find container \"c67a17a25be544d77a71bb2eb4e950f1e22793ec26be3689b50bbc58109f8987\": container with ID starting with c67a17a25be544d77a71bb2eb4e950f1e22793ec26be3689b50bbc58109f8987 not found: ID does not exist" Sep 16 01:47:03.547857 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.547815 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch6fd\" (UniqueName: \"kubernetes.io/projected/9774ea02-8641-4166-9c73-4551543fe5f3-kube-api-access-ch6fd\") pod \"9774ea02-8641-4166-9c73-4551543fe5f3\" (UID: \"9774ea02-8641-4166-9c73-4551543fe5f3\") " Sep 16 01:47:03.547857 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.547860 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9774ea02-8641-4166-9c73-4551543fe5f3-plugins-config-volume\") pod \"9774ea02-8641-4166-9c73-4551543fe5f3\" (UID: \"9774ea02-8641-4166-9c73-4551543fe5f3\") " Sep 16 01:47:03.548264 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.548241 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9774ea02-8641-4166-9c73-4551543fe5f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9774ea02-8641-4166-9c73-4551543fe5f3" (UID: "9774ea02-8641-4166-9c73-4551543fe5f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:47:03.550216 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.550187 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9774ea02-8641-4166-9c73-4551543fe5f3-kube-api-access-ch6fd" (OuterVolumeSpecName: "kube-api-access-ch6fd") pod "9774ea02-8641-4166-9c73-4551543fe5f3" (UID: "9774ea02-8641-4166-9c73-4551543fe5f3"). InnerVolumeSpecName "kube-api-access-ch6fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:47:03.649094 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.649070 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch6fd\" (UniqueName: \"kubernetes.io/projected/9774ea02-8641-4166-9c73-4551543fe5f3-kube-api-access-ch6fd\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:47:03.649094 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.649093 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9774ea02-8641-4166-9c73-4551543fe5f3-plugins-config-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:47:03.704664 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.704637 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd"] Sep 16 01:47:03.707870 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.707847 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c0ec73-96dcdfbdf-hhgkd"] Sep 16 01:47:03.737483 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:03.737459 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:47:04.296869 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:04.296829 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="125ca277-f2a2-4d69-ab6c-853b1dc09588" path="/var/lib/kubelet/pods/125ca277-f2a2-4d69-ab6c-853b1dc09588/volumes" Sep 16 01:47:04.297351 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:04.297186 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9774ea02-8641-4166-9c73-4551543fe5f3" path="/var/lib/kubelet/pods/9774ea02-8641-4166-9c73-4551543fe5f3/volumes" Sep 16 01:47:05.714465 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:05.714429 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:47:09.432043 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:09.432011 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:47:09.533567 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:09.533529 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:47:09.694944 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:09.694860 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:47:13.313945 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:13.313902 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:47:20.344738 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:20.344693 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:47:20.396042 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:20.396010 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:47:50.152977 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:50.152944 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:47:56.555179 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:56.555127 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:47:56.608655 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:47:56.608628 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:48:18.469769 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:48:18.469736 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:48:30.508759 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:48:30.508712 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:48:36.710022 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:48:36.709986 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:48:38.049147 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:48:38.049109 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:48:39.453326 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:48:39.453288 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:49:04.627405 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:04.627325 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:49:11.720986 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:11.720954 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:49:18.115742 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:18.115705 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:49:25.618754 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:25.618722 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:49:43.811050 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:43.811014 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:49:50.413512 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:50.413481 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:49:53.678959 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:53.678919 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24"] Sep 16 01:49:53.679459 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:53.679224 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" podUID="d2c98573-5b4a-4907-a4e8-9a13cbf37723" containerName="manager" containerID="cri-o://69d6d4b975f1653b44735d76a384a00536e4ac300ce4519d76b33c40364b3d43" gracePeriod=10 Sep 16 01:49:54.337821 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:54.337786 2558 generic.go:356] "Generic (PLEG): container finished" podID="d2c98573-5b4a-4907-a4e8-9a13cbf37723" containerID="69d6d4b975f1653b44735d76a384a00536e4ac300ce4519d76b33c40364b3d43" exitCode=0 Sep 16 01:49:54.337978 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:54.337829 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" event={"ID":"d2c98573-5b4a-4907-a4e8-9a13cbf37723","Type":"ContainerDied","Data":"69d6d4b975f1653b44735d76a384a00536e4ac300ce4519d76b33c40364b3d43"} Sep 16 01:49:54.436642 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:54.436618 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" Sep 16 01:49:54.534358 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:54.534277 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2c98573-5b4a-4907-a4e8-9a13cbf37723-extensions-socket-volume\") pod \"d2c98573-5b4a-4907-a4e8-9a13cbf37723\" (UID: \"d2c98573-5b4a-4907-a4e8-9a13cbf37723\") " Sep 16 01:49:54.534358 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:54.534321 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wf7f\" (UniqueName: \"kubernetes.io/projected/d2c98573-5b4a-4907-a4e8-9a13cbf37723-kube-api-access-9wf7f\") pod \"d2c98573-5b4a-4907-a4e8-9a13cbf37723\" (UID: \"d2c98573-5b4a-4907-a4e8-9a13cbf37723\") " Sep 16 01:49:54.534694 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:54.534667 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c98573-5b4a-4907-a4e8-9a13cbf37723-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d2c98573-5b4a-4907-a4e8-9a13cbf37723" (UID: "d2c98573-5b4a-4907-a4e8-9a13cbf37723"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:49:54.536444 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:54.536421 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c98573-5b4a-4907-a4e8-9a13cbf37723-kube-api-access-9wf7f" (OuterVolumeSpecName: "kube-api-access-9wf7f") pod "d2c98573-5b4a-4907-a4e8-9a13cbf37723" (UID: "d2c98573-5b4a-4907-a4e8-9a13cbf37723"). InnerVolumeSpecName "kube-api-access-9wf7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:49:54.635926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:54.635901 2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2c98573-5b4a-4907-a4e8-9a13cbf37723-extensions-socket-volume\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:49:54.635926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:54.635926 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wf7f\" (UniqueName: \"kubernetes.io/projected/d2c98573-5b4a-4907-a4e8-9a13cbf37723-kube-api-access-9wf7f\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:49:55.345941 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:55.345906 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" event={"ID":"d2c98573-5b4a-4907-a4e8-9a13cbf37723","Type":"ContainerDied","Data":"09f01786627f86f165b834fae99d24855aa658e896474b9fdfbcf86cba43a377"} Sep 16 01:49:55.345941 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:55.345929 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24" Sep 16 01:49:55.348514 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:55.345961 2558 scope.go:117] "RemoveContainer" containerID="69d6d4b975f1653b44735d76a384a00536e4ac300ce4519d76b33c40364b3d43" Sep 16 01:49:55.367126 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:55.367098 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24"] Sep 16 01:49:55.369944 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:55.369924 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sfj24"] Sep 16 01:49:56.294780 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:49:56.294745 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2c98573-5b4a-4907-a4e8-9a13cbf37723" path="/var/lib/kubelet/pods/d2c98573-5b4a-4907-a4e8-9a13cbf37723/volumes" Sep 16 01:50:26.791024 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:26.790991 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:50:54.500776 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.500734 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6"] Sep 16 01:50:54.501150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501093 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerName="istio-proxy" Sep 16 01:50:54.501150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501104 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerName="istio-proxy" Sep 16 01:50:54.501150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501112 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32fd6a87-8388-4175-b327-15dee2429085" containerName="payload-pre-processing" Sep 16 01:50:54.501150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501119 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="32fd6a87-8388-4175-b327-15dee2429085" containerName="payload-pre-processing" Sep 16 01:50:54.501150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501127 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c553b16-a983-46b2-ba04-a19cccebc22f" containerName="cleanup" Sep 16 01:50:54.501150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501133 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c553b16-a983-46b2-ba04-a19cccebc22f" containerName="cleanup" Sep 16 01:50:54.501150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501139 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8" containerName="payload-pre-processing" Sep 16 01:50:54.501150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501145 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8" containerName="payload-pre-processing" Sep 16 01:50:54.501150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501151 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerName="istio-proxy" Sep 16 01:50:54.501150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501173 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerName="istio-proxy" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501179 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerName="istio-proxy" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501184 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerName="istio-proxy" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501191 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9774ea02-8641-4166-9c73-4551543fe5f3" containerName="payload-processing" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501196 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9774ea02-8641-4166-9c73-4551543fe5f3" containerName="payload-processing" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501202 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerName="istio-proxy" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501209 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerName="istio-proxy" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501216 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5a3df4-818f-4c84-8187-c29a4ca5b7e3" containerName="payload-processing" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501221 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5a3df4-818f-4c84-8187-c29a4ca5b7e3" containerName="payload-processing" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501231 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00be5b2f-36e7-4eb1-8248-d722ca0e72ac" containerName="cleanup" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501236 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="00be5b2f-36e7-4eb1-8248-d722ca0e72ac" containerName="cleanup" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501242 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125ca277-f2a2-4d69-ab6c-853b1dc09588" containerName="cleanup" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501247 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="125ca277-f2a2-4d69-ab6c-853b1dc09588" containerName="cleanup" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501254 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c98573-5b4a-4907-a4e8-9a13cbf37723" containerName="manager" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501258 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c98573-5b4a-4907-a4e8-9a13cbf37723" containerName="manager" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501270 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e572b8a-1c4c-497f-8b11-195b92f33c0c" containerName="payload-processing" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501275 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e572b8a-1c4c-497f-8b11-195b92f33c0c" containerName="payload-processing" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501281 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5038efde-3484-4cda-80c3-482e794daf28" containerName="main" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501286 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5038efde-3484-4cda-80c3-482e794daf28" containerName="main" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501291 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerName="istio-proxy" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501297 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerName="istio-proxy" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501345 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9774ea02-8641-4166-9c73-4551543fe5f3" containerName="payload-processing" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501353 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="32fd6a87-8388-4175-b327-15dee2429085" containerName="payload-pre-processing" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501360 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e572b8a-1c4c-497f-8b11-195b92f33c0c" containerName="payload-processing" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501365 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerName="istio-proxy" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501372 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerName="istio-proxy" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501377 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="125ca277-f2a2-4d69-ab6c-853b1dc09588" containerName="cleanup" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501382 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc26edd-b9e5-41b5-a015-e2f9f63e2586" containerName="istio-proxy" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501389 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c98573-5b4a-4907-a4e8-9a13cbf37723" containerName="manager" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501394 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c44983e-f5c6-4c73-b8a1-844e47b3390e" containerName="istio-proxy" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501400 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="00be5b2f-36e7-4eb1-8248-d722ca0e72ac" containerName="cleanup" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501404 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5867dbc8-1438-4f0a-9d6d-a2c1d8fd1bb8" containerName="payload-pre-processing" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501410 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerName="istio-proxy" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501416 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5038efde-3484-4cda-80c3-482e794daf28" containerName="main" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501421 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerName="istio-proxy" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501426 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5a3df4-818f-4c84-8187-c29a4ca5b7e3" containerName="payload-processing" Sep 16 01:50:54.501492 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.501431 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c553b16-a983-46b2-ba04-a19cccebc22f" containerName="cleanup" Sep 16 01:50:54.504491 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.504475 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6" Sep 16 01:50:54.506739 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.506714 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-h9x8n\"" Sep 16 01:50:54.513867 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.513843 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6"] Sep 16 01:50:54.612251 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.612220 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526lr\" (UniqueName: \"kubernetes.io/projected/06b39685-bf27-4036-9962-769868a6e6f5-kube-api-access-526lr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rhbm6\" (UID: \"06b39685-bf27-4036-9962-769868a6e6f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6" Sep 16 01:50:54.612413 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.612305 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b39685-bf27-4036-9962-769868a6e6f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rhbm6\" (UID: \"06b39685-bf27-4036-9962-769868a6e6f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6" Sep 16 01:50:54.712692 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.712660 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526lr\" (UniqueName: \"kubernetes.io/projected/06b39685-bf27-4036-9962-769868a6e6f5-kube-api-access-526lr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rhbm6\" (UID: \"06b39685-bf27-4036-9962-769868a6e6f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6" Sep 16 01:50:54.712834 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.712728 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b39685-bf27-4036-9962-769868a6e6f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rhbm6\" (UID: \"06b39685-bf27-4036-9962-769868a6e6f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6" Sep 16 01:50:54.713076 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.713059 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b39685-bf27-4036-9962-769868a6e6f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rhbm6\" (UID: \"06b39685-bf27-4036-9962-769868a6e6f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6" Sep 16 01:50:54.720427 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.720403 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-526lr\" (UniqueName: \"kubernetes.io/projected/06b39685-bf27-4036-9962-769868a6e6f5-kube-api-access-526lr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rhbm6\" (UID: \"06b39685-bf27-4036-9962-769868a6e6f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6" Sep 16 01:50:54.815604 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:54.815545 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6" Sep 16 01:50:55.092125 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:55.091993 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6"] Sep 16 01:50:55.094331 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:50:55.094304 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b39685_bf27_4036_9962_769868a6e6f5.slice/crio-4fe57a5c9c230f39af5028c8afe742eb762e81b10803fac03e01bd1f9c134b1e WatchSource:0}: Error finding container 4fe57a5c9c230f39af5028c8afe742eb762e81b10803fac03e01bd1f9c134b1e: Status 404 returned error can't find the container with id 4fe57a5c9c230f39af5028c8afe742eb762e81b10803fac03e01bd1f9c134b1e Sep 16 01:50:55.673489 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:55.673454 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6" event={"ID":"06b39685-bf27-4036-9962-769868a6e6f5","Type":"ContainerStarted","Data":"2a85b4ebaed1a5a84368103277576075e2a8a03d9f549da2b8bd48639b8a1a54"} Sep 16 01:50:55.673901 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:55.673510 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6" event={"ID":"06b39685-bf27-4036-9962-769868a6e6f5","Type":"ContainerStarted","Data":"4fe57a5c9c230f39af5028c8afe742eb762e81b10803fac03e01bd1f9c134b1e"} Sep 16 01:50:55.673901 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:55.673547 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6" Sep 16 01:50:55.694115 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:50:55.694065 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6" podStartSLOduration=1.6940504889999999 podStartE2EDuration="1.694050489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:50:55.690795164 +0000 UTC m=+2085.943217754" watchObservedRunningTime="2026-09-16 01:50:55.694050489 +0000 UTC m=+2085.946473171" Sep 16 01:51:06.682100 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:06.682069 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbm6" Sep 16 01:51:10.420703 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:10.420675 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:51:10.432323 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:10.432300 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:51:10.723773 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:10.723703 2558 scope.go:117] "RemoveContainer" containerID="0b79b8d4df046793d8837b239dc1609de3c41aeb56ff1507d18262f6700dfe85" Sep 16 01:51:10.732202 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:10.732175 2558 scope.go:117] "RemoveContainer" containerID="68c963fdfaa231fad18b38bec8c80ec6e5300687af436655ab2b2f571ba576ab" Sep 16 01:51:18.118068 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:18.118031 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:51:19.006931 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:19.006895 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:51:19.108176 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:19.108128 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:51:24.612826 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:24.612785 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:51:29.706444 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:29.706402 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c"] Sep 16 01:51:29.706925 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:29.706906 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerName="istio-proxy" Sep 16 01:51:29.706925 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:29.706922 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32f1343-9c5c-4333-ba98-fbd9c80655c6" containerName="istio-proxy" Sep 16 01:51:29.710238 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:29.710218 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:29.712290 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:29.712273 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a084e1f3-kserve-self-signed-certs\"" Sep 16 01:51:29.719503 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:29.719480 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c"] Sep 16 01:51:29.909649 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:29.909617 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-dshm\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:29.909649 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:29.909649 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4l4w\" (UniqueName: \"kubernetes.io/projected/993ac568-d959-41a5-a3bf-8a9f1cfde894-kube-api-access-b4l4w\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:29.909876 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:29.909714 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-home\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:29.909876 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:29.909738 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993ac568-d959-41a5-a3bf-8a9f1cfde894-tls-certs\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:29.909876 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:29.909762 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-tmp-dir\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:29.909876 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:29.909836 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-model-cache\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.003146 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.003066 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:51:30.011213 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.011184 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-dshm\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.011355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.011222 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4l4w\" (UniqueName: \"kubernetes.io/projected/993ac568-d959-41a5-a3bf-8a9f1cfde894-kube-api-access-b4l4w\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.011355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.011283 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-home\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.011355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.011317 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993ac568-d959-41a5-a3bf-8a9f1cfde894-tls-certs\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.011355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.011341 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-tmp-dir\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.011572 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.011375 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-model-cache\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.011692 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.011665 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-home\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.011809 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.011788 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-tmp-dir\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.011872 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.011857 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-model-cache\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.013562 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.013536 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-dshm\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.013860 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.013839 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993ac568-d959-41a5-a3bf-8a9f1cfde894-tls-certs\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.017989 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.017964 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4l4w\" (UniqueName: \"kubernetes.io/projected/993ac568-d959-41a5-a3bf-8a9f1cfde894-kube-api-access-b4l4w\") pod \"e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.019655 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.019639 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:30.296879 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.296851 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c"] Sep 16 01:51:30.298735 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:51:30.298700 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod993ac568_d959_41a5_a3bf_8a9f1cfde894.slice/crio-bf1468bfb84de9220cb5735cf8290ded28f1c378ecd70085fb304cc8a909ad03 WatchSource:0}: Error finding container bf1468bfb84de9220cb5735cf8290ded28f1c378ecd70085fb304cc8a909ad03: Status 404 returned error can't find the container with id bf1468bfb84de9220cb5735cf8290ded28f1c378ecd70085fb304cc8a909ad03 Sep 16 01:51:30.880767 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.880725 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" event={"ID":"993ac568-d959-41a5-a3bf-8a9f1cfde894","Type":"ContainerStarted","Data":"95f3aaffc28ee6aa6e26317fb4d683fab6c729d0a648e9bab1271e55d8bc6d54"} Sep 16 01:51:30.881181 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.880777 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" event={"ID":"993ac568-d959-41a5-a3bf-8a9f1cfde894","Type":"ContainerStarted","Data":"bf1468bfb84de9220cb5735cf8290ded28f1c378ecd70085fb304cc8a909ad03"} Sep 16 01:51:30.899207 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:30.899150 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" podStartSLOduration=1.899136678 podStartE2EDuration="1.899136678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:51:30.896288401 +0000 UTC m=+2121.148711000" watchObservedRunningTime="2026-09-16 01:51:30.899136678 +0000 UTC m=+2121.151559268" Sep 16 01:51:32.916105 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:32.916069 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:51:38.339837 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:38.339806 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:51:40.020375 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:40.020344 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:40.020819 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:40.020391 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:40.032476 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:40.032451 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:40.943899 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:40.943870 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:51:42.819940 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:42.819908 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:51:52.969045 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:52.969009 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c"] Sep 16 01:51:52.988768 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:52.988726 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:51:54.013777 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:54.010900 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:51:54.527263 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:51:54.527224 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:52:02.609993 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:02.609914 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:52:07.989491 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:07.989447 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" podUID="993ac568-d959-41a5-a3bf-8a9f1cfde894" containerName="main" containerID="cri-o://95f3aaffc28ee6aa6e26317fb4d683fab6c729d0a648e9bab1271e55d8bc6d54" gracePeriod=45 Sep 16 01:52:08.243346 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.243294 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:52:08.298234 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.298187 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-dshm\") pod \"993ac568-d959-41a5-a3bf-8a9f1cfde894\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " Sep 16 01:52:08.298372 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.298246 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4l4w\" (UniqueName: \"kubernetes.io/projected/993ac568-d959-41a5-a3bf-8a9f1cfde894-kube-api-access-b4l4w\") pod \"993ac568-d959-41a5-a3bf-8a9f1cfde894\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " Sep 16 01:52:08.298372 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.298268 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-tmp-dir\") pod \"993ac568-d959-41a5-a3bf-8a9f1cfde894\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " Sep 16 01:52:08.298372 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.298305 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-model-cache\") pod \"993ac568-d959-41a5-a3bf-8a9f1cfde894\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " Sep 16 01:52:08.298502 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.298427 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993ac568-d959-41a5-a3bf-8a9f1cfde894-tls-certs\") pod \"993ac568-d959-41a5-a3bf-8a9f1cfde894\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " Sep 16 01:52:08.298568 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.298505 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-home\") pod \"993ac568-d959-41a5-a3bf-8a9f1cfde894\" (UID: \"993ac568-d959-41a5-a3bf-8a9f1cfde894\") " Sep 16 01:52:08.298568 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.298541 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "993ac568-d959-41a5-a3bf-8a9f1cfde894" (UID: "993ac568-d959-41a5-a3bf-8a9f1cfde894"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:52:08.298568 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.298555 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-model-cache" (OuterVolumeSpecName: "model-cache") pod "993ac568-d959-41a5-a3bf-8a9f1cfde894" (UID: "993ac568-d959-41a5-a3bf-8a9f1cfde894"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:52:08.298814 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.298794 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-tmp-dir\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:52:08.298814 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.298803 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-home" (OuterVolumeSpecName: "home") pod "993ac568-d959-41a5-a3bf-8a9f1cfde894" (UID: "993ac568-d959-41a5-a3bf-8a9f1cfde894"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:52:08.298814 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.298814 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-model-cache\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:52:08.300645 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.300619 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-dshm" (OuterVolumeSpecName: "dshm") pod "993ac568-d959-41a5-a3bf-8a9f1cfde894" (UID: "993ac568-d959-41a5-a3bf-8a9f1cfde894"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:52:08.300645 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.300627 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993ac568-d959-41a5-a3bf-8a9f1cfde894-kube-api-access-b4l4w" (OuterVolumeSpecName: "kube-api-access-b4l4w") pod "993ac568-d959-41a5-a3bf-8a9f1cfde894" (UID: "993ac568-d959-41a5-a3bf-8a9f1cfde894"). InnerVolumeSpecName "kube-api-access-b4l4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:52:08.300800 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.300717 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/993ac568-d959-41a5-a3bf-8a9f1cfde894-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "993ac568-d959-41a5-a3bf-8a9f1cfde894" (UID: "993ac568-d959-41a5-a3bf-8a9f1cfde894"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:52:08.399779 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.399749 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993ac568-d959-41a5-a3bf-8a9f1cfde894-tls-certs\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:52:08.399779 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.399776 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-home\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:52:08.399958 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.399788 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/993ac568-d959-41a5-a3bf-8a9f1cfde894-dshm\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:52:08.399958 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:08.399800 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4l4w\" (UniqueName: \"kubernetes.io/projected/993ac568-d959-41a5-a3bf-8a9f1cfde894-kube-api-access-b4l4w\") on node \"ip-10-0-137-55.ec2.internal\" DevicePath \"\"" Sep 16 01:52:09.097636 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:09.097604 2558 generic.go:356] "Generic (PLEG): container finished" podID="993ac568-d959-41a5-a3bf-8a9f1cfde894" containerID="95f3aaffc28ee6aa6e26317fb4d683fab6c729d0a648e9bab1271e55d8bc6d54" exitCode=0 Sep 16 01:52:09.098098 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:09.097648 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" Sep 16 01:52:09.098098 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:09.097684 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" event={"ID":"993ac568-d959-41a5-a3bf-8a9f1cfde894","Type":"ContainerDied","Data":"95f3aaffc28ee6aa6e26317fb4d683fab6c729d0a648e9bab1271e55d8bc6d54"} Sep 16 01:52:09.098098 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:09.097737 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c" event={"ID":"993ac568-d959-41a5-a3bf-8a9f1cfde894","Type":"ContainerDied","Data":"bf1468bfb84de9220cb5735cf8290ded28f1c378ecd70085fb304cc8a909ad03"} Sep 16 01:52:09.098098 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:09.097761 2558 scope.go:117] "RemoveContainer" containerID="95f3aaffc28ee6aa6e26317fb4d683fab6c729d0a648e9bab1271e55d8bc6d54" Sep 16 01:52:09.108889 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:09.108869 2558 scope.go:117] "RemoveContainer" containerID="95f3aaffc28ee6aa6e26317fb4d683fab6c729d0a648e9bab1271e55d8bc6d54" Sep 16 01:52:09.109139 ip-10-0-137-55 kubenswrapper[2558]: E0916 01:52:09.109120 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f3aaffc28ee6aa6e26317fb4d683fab6c729d0a648e9bab1271e55d8bc6d54\": container with ID starting with 95f3aaffc28ee6aa6e26317fb4d683fab6c729d0a648e9bab1271e55d8bc6d54 not found: ID does not exist" containerID="95f3aaffc28ee6aa6e26317fb4d683fab6c729d0a648e9bab1271e55d8bc6d54" Sep 16 01:52:09.109241 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:09.109147 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f3aaffc28ee6aa6e26317fb4d683fab6c729d0a648e9bab1271e55d8bc6d54"} err="failed to get container status \"95f3aaffc28ee6aa6e26317fb4d683fab6c729d0a648e9bab1271e55d8bc6d54\": rpc error: code = NotFound desc = could not find container \"95f3aaffc28ee6aa6e26317fb4d683fab6c729d0a648e9bab1271e55d8bc6d54\": container with ID starting with 95f3aaffc28ee6aa6e26317fb4d683fab6c729d0a648e9bab1271e55d8bc6d54 not found: ID does not exist" Sep 16 01:52:09.117962 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:09.117938 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c"] Sep 16 01:52:09.121193 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:09.121170 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a084e1f3-kserve-79f5ff6c7f-rmh9c"] Sep 16 01:52:10.301294 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:10.301260 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="993ac568-d959-41a5-a3bf-8a9f1cfde894" path="/var/lib/kubelet/pods/993ac568-d959-41a5-a3bf-8a9f1cfde894/volumes" Sep 16 01:52:10.616466 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:52:10.616433 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:55:16.919018 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:16.918982 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:55:23.721112 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:23.721078 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:55:29.610374 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:29.610341 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:55:36.818772 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:36.818736 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btqdv"] Sep 16 01:55:40.256423 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.256387 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hcspv/must-gather-ksxmq"] Sep 16 01:55:40.256814 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.256741 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="993ac568-d959-41a5-a3bf-8a9f1cfde894" containerName="main" Sep 16 01:55:40.256814 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.256751 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="993ac568-d959-41a5-a3bf-8a9f1cfde894" containerName="main" Sep 16 01:55:40.256891 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.256838 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="993ac568-d959-41a5-a3bf-8a9f1cfde894" containerName="main" Sep 16 01:55:40.260328 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.260303 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hcspv/must-gather-ksxmq" Sep 16 01:55:40.262696 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.262674 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hcspv\"/\"default-dockercfg-dfgwp\"" Sep 16 01:55:40.262838 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.262705 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hcspv\"/\"openshift-service-ca.crt\"" Sep 16 01:55:40.263491 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.263464 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hcspv\"/\"kube-root-ca.crt\"" Sep 16 01:55:40.274126 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.274106 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hcspv/must-gather-ksxmq"] Sep 16 01:55:40.301081 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.301051 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:55:40.385838 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.385807 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" limit=1024 Sep 16 01:55:40.422468 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.422442 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbshp\" (UniqueName: \"kubernetes.io/projected/8b102b6f-b573-477a-9086-c06a9ca71c8f-kube-api-access-wbshp\") pod \"must-gather-ksxmq\" (UID: \"8b102b6f-b573-477a-9086-c06a9ca71c8f\") " pod="openshift-must-gather-hcspv/must-gather-ksxmq" Sep 16 01:55:40.422572 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.422484 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b102b6f-b573-477a-9086-c06a9ca71c8f-must-gather-output\") pod \"must-gather-ksxmq\" (UID: \"8b102b6f-b573-477a-9086-c06a9ca71c8f\") " pod="openshift-must-gather-hcspv/must-gather-ksxmq" Sep 16 01:55:40.523781 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.523706 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b102b6f-b573-477a-9086-c06a9ca71c8f-must-gather-output\") pod \"must-gather-ksxmq\" (UID: \"8b102b6f-b573-477a-9086-c06a9ca71c8f\") " pod="openshift-must-gather-hcspv/must-gather-ksxmq" Sep 16 01:55:40.523919 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.523801 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbshp\" (UniqueName: \"kubernetes.io/projected/8b102b6f-b573-477a-9086-c06a9ca71c8f-kube-api-access-wbshp\") pod \"must-gather-ksxmq\" (UID: \"8b102b6f-b573-477a-9086-c06a9ca71c8f\") " pod="openshift-must-gather-hcspv/must-gather-ksxmq" Sep 16 01:55:40.524111 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.524086 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b102b6f-b573-477a-9086-c06a9ca71c8f-must-gather-output\") pod \"must-gather-ksxmq\" (UID: \"8b102b6f-b573-477a-9086-c06a9ca71c8f\") " pod="openshift-must-gather-hcspv/must-gather-ksxmq" Sep 16 01:55:40.531472 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.531448 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbshp\" (UniqueName: \"kubernetes.io/projected/8b102b6f-b573-477a-9086-c06a9ca71c8f-kube-api-access-wbshp\") pod \"must-gather-ksxmq\" (UID: \"8b102b6f-b573-477a-9086-c06a9ca71c8f\") " pod="openshift-must-gather-hcspv/must-gather-ksxmq" Sep 16 01:55:40.570203 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.570177 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hcspv/must-gather-ksxmq" Sep 16 01:55:40.839978 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.839897 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hcspv/must-gather-ksxmq"] Sep 16 01:55:40.843734 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:55:40.843706 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b102b6f_b573_477a_9086_c06a9ca71c8f.slice/crio-fd78dc215b548ccb755a88618feb9ca09856775827676291f2fa9d70331ab6c8 WatchSource:0}: Error finding container fd78dc215b548ccb755a88618feb9ca09856775827676291f2fa9d70331ab6c8: Status 404 returned error can't find the container with id fd78dc215b548ccb755a88618feb9ca09856775827676291f2fa9d70331ab6c8 Sep 16 01:55:40.845751 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:40.845733 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 01:55:41.247276 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:41.247242 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hcspv/must-gather-ksxmq" event={"ID":"8b102b6f-b573-477a-9086-c06a9ca71c8f","Type":"ContainerStarted","Data":"fd78dc215b548ccb755a88618feb9ca09856775827676291f2fa9d70331ab6c8"} Sep 16 01:55:42.257658 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:42.257622 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hcspv/must-gather-ksxmq" event={"ID":"8b102b6f-b573-477a-9086-c06a9ca71c8f","Type":"ContainerStarted","Data":"491bd31def60ac2305dc5e0415fbdcf2f58207bc55672279e5e06707b89ae636"} Sep 16 01:55:42.258150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:42.257671 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hcspv/must-gather-ksxmq" event={"ID":"8b102b6f-b573-477a-9086-c06a9ca71c8f","Type":"ContainerStarted","Data":"4c1a8bc53283e86748fa96eab40d57df1ce90e88784d5c9564f9c5e6590b6d8f"} Sep 16 01:55:42.282459 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:42.282389 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hcspv/must-gather-ksxmq" podStartSLOduration=1.468128236 podStartE2EDuration="2.282367941s" totalImagesPullingTime="814.239705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:55:42.280514199 +0000 UTC m=+2372.532936791" watchObservedRunningTime="2026-09-16 01:55:42.282367941 +0000 UTC m=+2372.534790532" Sep 16 01:55:43.156926 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:43.156890 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6ndbn_5826c82e-35c3-40b5-a3f8-774437001cf8/global-pull-secret-syncer/0.log" Sep 16 01:55:43.304108 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:43.304076 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-49bsb_86bd0cbd-70d8-47c2-b865-836ff085681a/konnectivity-agent/0.log" Sep 16 01:55:43.393481 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:43.393450 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-55.ec2.internal_1e7172446b6553eb9092c5ca4da52a1b/haproxy/0.log" Sep 16 01:55:47.866741 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:47.866714 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-v5wjf_023e4195-ebc3-4f80-8e7b-d7ac7e145e56/manager/0.log" Sep 16 01:55:47.926590 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:47.926551 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-rhbm6_06b39685-bf27-4036-9962-769868a6e6f5/manager/0.log" Sep 16 01:55:47.947111 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:47.947062 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-btqdv_5b2e403b-f3b8-46a9-b893-40a74db00bf8/limitador/0.log" Sep 16 01:55:49.869201 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:49.869149 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-gcbb4_e7b745db-d94b-428e-8012-cab5a3d44d1d/kube-state-metrics/0.log" Sep 16 01:55:49.882851 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:49.882818 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-gcbb4_e7b745db-d94b-428e-8012-cab5a3d44d1d/kube-rbac-proxy-main/0.log" Sep 16 01:55:49.902922 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:49.902886 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-gcbb4_e7b745db-d94b-428e-8012-cab5a3d44d1d/kube-rbac-proxy-self/0.log" Sep 16 01:55:49.927432 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:49.927391 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6496c56f47-t8xsm_ad694c3c-8319-4a83-a520-a6a13c1a077e/metrics-server/0.log" Sep 16 01:55:50.049014 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:50.048979 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh6bv_af3a00ef-4f24-41c6-8ec5-65f9fc8685ac/node-exporter/0.log" Sep 16 01:55:50.069488 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:50.069462 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh6bv_af3a00ef-4f24-41c6-8ec5-65f9fc8685ac/kube-rbac-proxy/0.log" Sep 16 01:55:50.100013 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:50.099983 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh6bv_af3a00ef-4f24-41c6-8ec5-65f9fc8685ac/init-textfile/0.log" Sep 16 01:55:50.457670 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:50.457639 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-x9kmd_2ca52899-0cc2-4d38-942f-3659d1113017/prometheus-operator-admission-webhook/0.log" Sep 16 01:55:50.556964 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:50.556863 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4c7d77c7-wjvsp_a734a692-d38a-4b2b-b932-de84fea3dceb/thanos-query/0.log" Sep 16 01:55:50.576855 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:50.576819 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4c7d77c7-wjvsp_a734a692-d38a-4b2b-b932-de84fea3dceb/kube-rbac-proxy-web/0.log" Sep 16 01:55:50.594523 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:50.594492 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4c7d77c7-wjvsp_a734a692-d38a-4b2b-b932-de84fea3dceb/kube-rbac-proxy/0.log" Sep 16 01:55:50.614800 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:50.614773 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4c7d77c7-wjvsp_a734a692-d38a-4b2b-b932-de84fea3dceb/prom-label-proxy/0.log" Sep 16 01:55:50.633288 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:50.633258 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4c7d77c7-wjvsp_a734a692-d38a-4b2b-b932-de84fea3dceb/kube-rbac-proxy-rules/0.log" Sep 16 01:55:50.659632 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:50.659600 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4c7d77c7-wjvsp_a734a692-d38a-4b2b-b932-de84fea3dceb/kube-rbac-proxy-metrics/0.log" Sep 16 01:55:52.824066 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:52.824039 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:55:52.833148 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:52.833095 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/2.log" Sep 16 01:55:53.410557 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:53.410532 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c8c848f8f-plcg9_d114a491-470a-408f-b155-e8ae4d514341/console/0.log" Sep 16 01:55:54.022306 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.022264 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76"] Sep 16 01:55:54.028383 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.028356 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.037526 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.037495 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76"] Sep 16 01:55:54.072950 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.072901 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed495e20-fd54-4b07-a45d-764fad72939f-lib-modules\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.072950 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.072952 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ed495e20-fd54-4b07-a45d-764fad72939f-proc\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.073229 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.072978 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ed495e20-fd54-4b07-a45d-764fad72939f-podres\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.073229 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.073042 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed495e20-fd54-4b07-a45d-764fad72939f-sys\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.073229 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.073121 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kks2j\" (UniqueName: \"kubernetes.io/projected/ed495e20-fd54-4b07-a45d-764fad72939f-kube-api-access-kks2j\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.173797 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.173749 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed495e20-fd54-4b07-a45d-764fad72939f-lib-modules\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.173988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.173814 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ed495e20-fd54-4b07-a45d-764fad72939f-proc\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.173988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.173843 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ed495e20-fd54-4b07-a45d-764fad72939f-podres\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.173988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.173888 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed495e20-fd54-4b07-a45d-764fad72939f-sys\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.173988 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.173973 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kks2j\" (UniqueName: \"kubernetes.io/projected/ed495e20-fd54-4b07-a45d-764fad72939f-kube-api-access-kks2j\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.174545 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.174515 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed495e20-fd54-4b07-a45d-764fad72939f-lib-modules\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.174618 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.174600 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ed495e20-fd54-4b07-a45d-764fad72939f-proc\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.174702 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.174685 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ed495e20-fd54-4b07-a45d-764fad72939f-podres\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.174763 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.174742 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed495e20-fd54-4b07-a45d-764fad72939f-sys\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.182098 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.182068 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kks2j\" (UniqueName: \"kubernetes.io/projected/ed495e20-fd54-4b07-a45d-764fad72939f-kube-api-access-kks2j\") pod \"perf-node-gather-daemonset-xdh76\" (UID: \"ed495e20-fd54-4b07-a45d-764fad72939f\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.345128 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.345028 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:54.639057 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:54.639023 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76"] Sep 16 01:55:54.644419 ip-10-0-137-55 kubenswrapper[2558]: W0916 01:55:54.644381 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poded495e20_fd54_4b07_a45d_764fad72939f.slice/crio-433cf333764225644dad7c99cbb3f88fe3f9a450bea854f5430a7bea7b41ca47 WatchSource:0}: Error finding container 433cf333764225644dad7c99cbb3f88fe3f9a450bea854f5430a7bea7b41ca47: Status 404 returned error can't find the container with id 433cf333764225644dad7c99cbb3f88fe3f9a450bea854f5430a7bea7b41ca47 Sep 16 01:55:55.287304 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:55.287276 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lnb9f_36024d3d-ea39-4326-b503-4d8a07691efa/dns/0.log" Sep 16 01:55:55.305892 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:55.305867 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lnb9f_36024d3d-ea39-4326-b503-4d8a07691efa/kube-rbac-proxy/0.log" Sep 16 01:55:55.373848 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:55.373814 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" event={"ID":"ed495e20-fd54-4b07-a45d-764fad72939f","Type":"ContainerStarted","Data":"dcf04585f307ff0b39e7f382670eeeb9776e14d6e97174c4acdcdd89c5159af5"} Sep 16 01:55:55.374039 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:55.373873 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" event={"ID":"ed495e20-fd54-4b07-a45d-764fad72939f","Type":"ContainerStarted","Data":"433cf333764225644dad7c99cbb3f88fe3f9a450bea854f5430a7bea7b41ca47"} Sep 16 01:55:55.374039 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:55.373899 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:55:55.389032 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:55.388979 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" podStartSLOduration=1.38896262 podStartE2EDuration="1.38896262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:55:55.386615725 +0000 UTC m=+2385.639038309" watchObservedRunningTime="2026-09-16 01:55:55.38896262 +0000 UTC m=+2385.641385238" Sep 16 01:55:55.390295 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:55.390269 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9twcf_f6be6ffd-8560-4250-84a4-0741806d4858/dns-node-resolver/0.log" Sep 16 01:55:55.921648 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:55.921606 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-659c7b6595-x87bp_04040d3e-3b48-47f2-8137-55ffd432390d/registry/0.log" Sep 16 01:55:55.943324 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:55.943292 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8s8qx_2a63a0c6-8521-4839-9955-1d4f7e6ea1a9/node-ca/0.log" Sep 16 01:55:57.258972 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:57.258918 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7tf4c_cf21b222-a313-4498-8f0c-016d7fe3656a/istio-proxy/0.log" Sep 16 01:55:57.265003 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:57.264977 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7tf4c_cf21b222-a313-4498-8f0c-016d7fe3656a/istio-proxy/1.log" Sep 16 01:55:58.186375 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:58.186344 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zczs4_0974aa23-1b99-46bd-a1b0-fdc7f7657069/serve-healthcheck-canary/0.log" Sep 16 01:55:58.633998 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:58.633958 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-dgbrm_d157e624-20f1-4d08-a7aa-23a3b0b1d190/insights-operator/0.log" Sep 16 01:55:58.638709 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:58.638677 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-dgbrm_d157e624-20f1-4d08-a7aa-23a3b0b1d190/insights-operator/1.log" Sep 16 01:55:58.777228 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:58.777199 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rrxxj_9633a4e7-63e6-44e1-9469-7334c5dc56e4/kube-rbac-proxy/0.log" Sep 16 01:55:58.795369 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:58.795336 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rrxxj_9633a4e7-63e6-44e1-9469-7334c5dc56e4/exporter/0.log" Sep 16 01:55:58.815862 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:55:58.815811 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rrxxj_9633a4e7-63e6-44e1-9469-7334c5dc56e4/extractor/0.log" Sep 16 01:56:00.857150 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:00.857116 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-q42rb_63ff281e-5d24-44a2-970c-a6ef19b8f64b/manager/0.log" Sep 16 01:56:01.266174 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:01.266108 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-skkpk_8b44e0e4-9e34-4811-a629-0516078a58a1/manager/0.log" Sep 16 01:56:01.328202 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:01.328171 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mpg47_5252e61c-a46e-42c4-a1b5-d2f2fd9528ac/server/0.log" Sep 16 01:56:01.396393 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:01.396365 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-xdh76" Sep 16 01:56:01.471417 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:01.471380 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lkdhm_e0926020-5dbf-41d5-b6e0-21a08053d587/manager/0.log" Sep 16 01:56:01.534989 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:01.534906 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f84c66984-k7wxp_01611d7f-9833-4b82-986d-61ce785f0a94/manager/0.log" Sep 16 01:56:01.555296 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:01.555266 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f84c66984-k7wxp_01611d7f-9833-4b82-986d-61ce785f0a94/copy-manifests/0.log" Sep 16 01:56:02.938116 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:02.938038 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-65dff4dd97-b967m_29412d1b-b3cb-45e9-b326-53469931bf2d/manager/0.log" Sep 16 01:56:04.420531 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:04.420501 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7tf4c_cf21b222-a313-4498-8f0c-016d7fe3656a/istio-proxy/0.log" Sep 16 01:56:04.426091 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:04.426048 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7tf4c_cf21b222-a313-4498-8f0c-016d7fe3656a/istio-proxy/1.log" Sep 16 01:56:06.382751 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:06.382710 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7tf4c_cf21b222-a313-4498-8f0c-016d7fe3656a/istio-proxy/0.log" Sep 16 01:56:06.387884 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:06.387858 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7tf4c_cf21b222-a313-4498-8f0c-016d7fe3656a/istio-proxy/1.log" Sep 16 01:56:08.088188 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:08.088131 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-gwbth_6f156ed2-e2d7-4711-abb7-91765eca5f53/migrator/0.log" Sep 16 01:56:08.105434 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:08.105407 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-gwbth_6f156ed2-e2d7-4711-abb7-91765eca5f53/graceful-termination/0.log" Sep 16 01:56:09.476533 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:09.476501 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjqv9_0b2afed4-62a6-41d3-b80b-61e37294be11/kube-multus-additional-cni-plugins/0.log" Sep 16 01:56:09.494592 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:09.494568 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjqv9_0b2afed4-62a6-41d3-b80b-61e37294be11/egress-router-binary-copy/0.log" Sep 16 01:56:09.512614 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:09.512589 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjqv9_0b2afed4-62a6-41d3-b80b-61e37294be11/cni-plugins/0.log" Sep 16 01:56:09.529734 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:09.529710 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjqv9_0b2afed4-62a6-41d3-b80b-61e37294be11/bond-cni-plugin/0.log" Sep 16 01:56:09.548300 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:09.548277 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjqv9_0b2afed4-62a6-41d3-b80b-61e37294be11/routeoverride-cni/0.log" Sep 16 01:56:09.566655 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:09.566632 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjqv9_0b2afed4-62a6-41d3-b80b-61e37294be11/whereabouts-cni-bincopy/0.log" Sep 16 01:56:09.583479 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:09.583457 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjqv9_0b2afed4-62a6-41d3-b80b-61e37294be11/whereabouts-cni/0.log" Sep 16 01:56:09.881000 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:09.880965 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rq7cg_30ce019d-bb0b-45ff-a742-cf9c8f44efe1/kube-multus/0.log" Sep 16 01:56:09.906001 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:09.905969 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hs6d5_2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c/network-metrics-daemon/0.log" Sep 16 01:56:09.921001 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:09.920972 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hs6d5_2cc1e9be-9d70-4ed6-9894-c7b7342a0c0c/kube-rbac-proxy/0.log" Sep 16 01:56:10.478886 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:10.478851 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:56:10.490508 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:10.490481 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2gmqp_9d8685e8-58d2-439a-9879-37eeb804efa2/console-operator/1.log" Sep 16 01:56:11.191346 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:11.191258 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mgv65_db66eb80-3fa5-4dee-9ba7-c18ba216ab4c/ovn-controller/0.log" Sep 16 01:56:11.204677 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:11.204655 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mgv65_db66eb80-3fa5-4dee-9ba7-c18ba216ab4c/ovn-acl-logging/0.log" Sep 16 01:56:11.225920 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:11.225899 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mgv65_db66eb80-3fa5-4dee-9ba7-c18ba216ab4c/kube-rbac-proxy-node/0.log" Sep 16 01:56:11.243327 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:11.243292 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mgv65_db66eb80-3fa5-4dee-9ba7-c18ba216ab4c/kube-rbac-proxy-ovn-metrics/0.log" Sep 16 01:56:11.257534 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:11.257515 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mgv65_db66eb80-3fa5-4dee-9ba7-c18ba216ab4c/northd/0.log" Sep 16 01:56:11.274823 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:11.274759 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mgv65_db66eb80-3fa5-4dee-9ba7-c18ba216ab4c/nbdb/0.log" Sep 16 01:56:11.293355 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:11.293311 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mgv65_db66eb80-3fa5-4dee-9ba7-c18ba216ab4c/sbdb/0.log" Sep 16 01:56:11.390181 ip-10-0-137-55 kubenswrapper[2558]: I0916 01:56:11.389236 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mgv65_db66eb80-3fa5-4dee-9ba7-c18ba216ab4c/ovnkube-controller/0.log"