Jul 10 16:54:19.745800 ip-10-0-141-228 systemd[1]: Starting Kubernetes Kubelet... Jul 10 16:54:20.159769 ip-10-0-141-228 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 16:54:20.159769 ip-10-0-141-228 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 16:54:20.159769 ip-10-0-141-228 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 16:54:20.159769 ip-10-0-141-228 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 16:54:20.159769 ip-10-0-141-228 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 16:54:20.160802 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.160711 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 16:54:20.166945 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166920 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:54:20.166945 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166941 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:54:20.166945 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166945 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:54:20.166945 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166948 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:54:20.166945 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166952 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:54:20.166945 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166956 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166959 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166962 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166965 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166968 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166970 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166973 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166975 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166978 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166981 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166983 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166986 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166989 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166991 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.166997 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167000 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167003 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167007 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167011 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167014 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:54:20.167168 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167017 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167020 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167023 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167026 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167030 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167033 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167036 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167038 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167041 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167043 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167046 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167049 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167052 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167061 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167063 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167066 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167069 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167071 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167074 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:54:20.167695 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167077 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167079 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167082 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167084 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167089 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167092 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167096 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167099 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167102 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167105 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167107 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167110 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167112 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167115 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167117 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167120 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167122 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167125 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167128 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167130 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:54:20.168135 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167133 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167135 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167137 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167142 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167147 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167150 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167154 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167157 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167160 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167162 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167165 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167167 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167170 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167173 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167175 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167178 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167180 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167183 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167186 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167189 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:54:20.168649 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167191 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167194 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167197 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167199 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167201 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167204 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167206 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167209 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167212 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167214 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167216 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167219 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167221 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167224 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.167226 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.168959 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.168968 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.168973 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.168976 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.168979 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:54:20.169127 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.168981 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.168984 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.168988 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.168992 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.168995 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.168998 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169001 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169004 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169006 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169009 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169011 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169014 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169017 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169020 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169023 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169026 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169029 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169031 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169034 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169036 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:54:20.169651 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169039 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169041 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169044 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169046 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169049 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169051 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169055 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169060 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169062 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169065 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169069 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169073 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169076 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169078 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169081 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169083 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169086 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169088 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169091 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:54:20.170187 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169093 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169095 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169097 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169100 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169102 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169105 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169108 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169111 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169113 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169116 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169119 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169121 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169124 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169126 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169129 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169131 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169134 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169136 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169139 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169143 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:54:20.170673 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169146 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169149 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169151 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169154 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169156 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169158 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169161 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169163 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169166 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169168 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169170 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169173 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169175 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169178 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169180 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169183 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169185 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169188 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169190 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169194 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:54:20.171163 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169197 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169200 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169202 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169205 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169207 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169210 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169212 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169214 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169217 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169219 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169222 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169224 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169227 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169230 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169232 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169634 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169639 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169642 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169645 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169647 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:54:20.171668 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169650 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169653 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169656 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169659 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169662 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169664 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169667 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169669 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169672 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169674 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169677 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169680 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169683 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169686 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169688 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169691 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169694 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169696 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169699 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:54:20.172143 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169703 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169707 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169711 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169713 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169717 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169720 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169729 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169732 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169735 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169737 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169741 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169743 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169746 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169748 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169751 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169754 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169756 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169759 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169762 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:54:20.172718 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169764 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169767 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169769 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169771 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169774 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169777 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169780 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169782 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169786 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169790 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169793 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169796 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169799 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169801 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169803 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169806 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169809 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169813 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169816 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:54:20.173181 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169818 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169821 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169824 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169826 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169829 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169831 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169834 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169836 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169839 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169842 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169845 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169847 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169849 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169852 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169854 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169857 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169859 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169861 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169864 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169866 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:54:20.173654 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169869 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169872 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169874 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169877 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169879 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169881 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169884 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169886 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169888 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169894 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169897 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169900 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169902 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169905 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169908 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169910 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.169913 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.169984 2559 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.169991 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.169998 2559 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170002 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 16:54:20.174131 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170006 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170009 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170013 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170017 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170020 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170023 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170027 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170030 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170033 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170036 2559 flags.go:64] FLAG: --cgroup-root="" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170038 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170041 2559 flags.go:64] FLAG: --client-ca-file="" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170045 2559 flags.go:64] FLAG: --cloud-provider="external" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170047 2559 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170052 2559 flags.go:64] FLAG: --cluster-domain="" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170054 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170057 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170060 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170064 2559 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170068 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170073 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170076 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170080 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170082 2559 flags.go:64] FLAG: --contention-profiling="false" Jul 10 16:54:20.174669 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170085 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170088 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170091 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170094 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170098 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170101 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170104 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170107 2559 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170109 2559 flags.go:64] FLAG: --enable-server="true" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170112 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170117 2559 flags.go:64] FLAG: --event-burst="100" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170120 2559 flags.go:64] FLAG: --event-qps="50" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170123 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170126 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170129 2559 flags.go:64] FLAG: --eviction-hard="" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170137 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170139 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170142 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170145 2559 flags.go:64] FLAG: --eviction-soft="" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170148 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170151 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170154 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170157 2559 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170159 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170162 2559 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 16:54:20.175255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170165 2559 flags.go:64] FLAG: --feature-gates="" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170169 2559 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170171 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170177 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170180 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170183 2559 flags.go:64] FLAG: --healthz-port="10248" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170186 2559 flags.go:64] FLAG: --help="false" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170189 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170193 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170196 2559 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170198 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170202 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170205 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170208 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170210 2559 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170213 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170216 2559 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170219 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170222 2559 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170224 2559 flags.go:64] FLAG: --kube-reserved="" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170227 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170230 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170233 2559 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170235 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 16:54:20.175870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170238 2559 flags.go:64] FLAG: --lock-file="" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170241 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170243 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170246 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170252 2559 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170256 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170258 2559 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170261 2559 flags.go:64] FLAG: --logging-format="text" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170264 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170267 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170269 2559 flags.go:64] FLAG: --manifest-url="" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170274 2559 flags.go:64] FLAG: --manifest-url-header="" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170278 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170281 2559 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170284 2559 flags.go:64] FLAG: --max-pods="110" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170287 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170290 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170293 2559 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170296 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170299 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170301 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170304 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170311 2559 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170314 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170317 2559 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 16:54:20.176443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170320 2559 flags.go:64] FLAG: --pod-cidr="" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170322 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170328 2559 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170331 2559 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170334 2559 flags.go:64] FLAG: --pods-per-core="0" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170336 2559 flags.go:64] FLAG: --port="10250" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170339 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170342 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08ff80ce5f4ca7bbd" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170345 2559 flags.go:64] FLAG: --qos-reserved="" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170348 2559 flags.go:64] FLAG: --read-only-port="10255" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170350 2559 flags.go:64] FLAG: --register-node="true" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170353 2559 flags.go:64] FLAG: --register-with-taints="" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170357 2559 flags.go:64] FLAG: --registry-burst="10" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170361 2559 flags.go:64] FLAG: --registry-qps="5" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170364 2559 flags.go:64] FLAG: --reserved-cpus="" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170366 2559 flags.go:64] FLAG: --reserved-memory="" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170369 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170372 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170376 2559 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170379 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170382 2559 flags.go:64] FLAG: --runonce="false" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170385 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170388 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170390 2559 flags.go:64] FLAG: --seccomp-default="false" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170393 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 16:54:20.177065 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170396 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170398 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170401 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170404 2559 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170406 2559 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170409 2559 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170412 2559 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170415 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170417 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170420 2559 flags.go:64] FLAG: --system-cgroups="" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170423 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170429 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170431 2559 flags.go:64] FLAG: --tls-cert-file="" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170434 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170438 2559 flags.go:64] FLAG: --tls-min-version="" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170441 2559 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170443 2559 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170446 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170449 2559 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170452 2559 flags.go:64] FLAG: --v="2" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170455 2559 flags.go:64] FLAG: --version="false" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170460 2559 flags.go:64] FLAG: --vmodule="" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170464 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.170467 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170597 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:54:20.177825 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170602 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170606 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170608 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170611 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170613 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170616 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170618 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170621 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170623 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170625 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170628 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170630 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170633 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170635 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170638 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170640 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170643 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170645 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170648 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170650 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:54:20.178941 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170652 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170655 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170657 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170660 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170662 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170664 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170667 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170669 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170671 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170675 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170677 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170680 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170684 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170688 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170691 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170694 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170697 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170700 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170703 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:54:20.179811 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170706 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170708 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170711 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170713 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170716 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170718 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170720 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170723 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170725 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170728 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170730 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170732 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170735 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170744 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170748 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170751 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170754 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170756 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170759 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170761 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:54:20.180709 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170764 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170766 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170770 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170773 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170776 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170779 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170781 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170784 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170786 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170789 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170791 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170793 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170796 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170798 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170801 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170803 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170805 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170808 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170810 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170813 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:54:20.181593 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170816 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170819 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170823 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170825 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170828 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170830 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170834 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170837 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170840 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170842 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170845 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170848 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170851 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170853 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170857 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170860 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170863 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170866 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:54:20.182161 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.170869 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.171520 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.180194 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.180214 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.180241 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.180254 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180340 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180348 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180353 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180358 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180363 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180368 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180372 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180377 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:54:20.182935 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180381 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180385 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180390 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180394 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180398 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180402 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180407 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180411 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180415 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180419 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180424 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180428 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180432 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180437 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180441 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180445 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180449 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180453 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180457 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180464 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:54:20.183547 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180470 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180477 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180482 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180487 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180510 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180514 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180519 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180523 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180527 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180531 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180535 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180539 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180543 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180547 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180551 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180555 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180559 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180563 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180568 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180571 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:54:20.184115 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180576 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180581 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180586 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180591 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180595 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180599 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180603 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180607 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180611 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180618 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180623 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180627 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180631 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180636 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180642 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180647 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180654 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180659 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180664 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:54:20.184837 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180669 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180675 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180679 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180683 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180687 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180692 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180696 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180701 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180705 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180709 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180713 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180716 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180720 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180725 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180728 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180733 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180737 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180741 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180746 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180750 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:54:20.185318 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180755 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180759 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180763 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180767 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180772 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180776 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180781 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180785 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180790 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180795 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180799 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.180803 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.180811 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181026 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181036 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:54:20.185851 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181040 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181044 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181049 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181053 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181057 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181061 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181066 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181070 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181074 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181078 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181084 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181091 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181095 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181099 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181104 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181108 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181113 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181117 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181121 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:54:20.186216 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181125 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181129 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181133 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181137 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181142 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181146 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181150 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181155 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181160 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181164 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181168 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181172 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181176 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181180 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181185 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181189 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181193 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181197 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181201 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:54:20.186756 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181208 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181213 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181218 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181223 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181227 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181231 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181235 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181239 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181243 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181247 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181251 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181256 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181260 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181264 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181268 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181272 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181276 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181280 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181285 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181289 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:54:20.187215 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181293 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181298 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181303 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181307 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181311 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181314 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181319 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181323 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181327 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181331 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181335 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181339 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181343 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181347 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181351 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181355 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181359 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181363 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181367 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181371 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:54:20.187778 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181376 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181380 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181384 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181389 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181393 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181397 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181401 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181405 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181409 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181414 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181418 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181422 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181426 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181430 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181435 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181439 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181445 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181450 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:54:20.188282 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:20.181454 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:54:20.188763 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.181461 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 16:54:20.188763 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.181643 2559 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 16:54:20.188763 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.184326 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 16:54:20.188763 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.186556 2559 server.go:1019] "Starting client certificate rotation" Jul 10 16:54:20.188763 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.186664 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 16:54:20.188763 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.186698 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 16:54:20.189303 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.189287 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 16:54:20.192539 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.192520 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 16:54:20.204579 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.204556 2559 log.go:25] "Validated CRI v1 runtime API" Jul 10 16:54:20.206551 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.206529 2559 log.go:25] "Validated CRI v1 image API" Jul 10 16:54:20.207342 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.207322 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 16:54:20.209564 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.209542 2559 fs.go:135] Filesystem UUIDs: map[3bb9a840-73d6-4b3c-adb5-05a04d41edd9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e8acac1d-9424-4249-9af1-5d1107859050:/dev/nvme0n1p4] Jul 10 16:54:20.209648 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.209562 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 16:54:20.212791 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.212775 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 16:54:20.215934 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.215821 2559 manager.go:216] Machine: {Timestamp:2026-07-10 16:54:20.214404683 +0000 UTC m=+0.340113084 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099554 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26755e32588bb39d76772c5d2b880f SystemUUID:ec26755e-3258-8bb3-9d76-772c5d2b880f BootID:2d90d2ea-eba2-4fbe-aa80-d0793eb25f9f Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6a:71:92:e5:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6a:71:92:e5:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:61:75:49:ea:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 16:54:20.215934 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.215928 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 16:54:20.216038 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.216010 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 16:54:20.219673 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.219652 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 16:54:20.219816 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.219678 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-228.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 16:54:20.220950 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.220941 2559 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 16:54:20.220980 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.220954 2559 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 16:54:20.220980 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.220962 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 16:54:20.221745 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.221735 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 16:54:20.223089 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.223078 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 16:54:20.226830 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.226819 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 10 16:54:20.226946 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.226936 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 16:54:20.226984 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.226968 2559 kubelet.go:486] "Attempting to sync node with API server" Jul 10 16:54:20.226984 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.226982 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 16:54:20.227060 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.226997 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 16:54:20.227060 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.227006 2559 kubelet.go:398] "Adding apiserver pod source" Jul 10 16:54:20.227060 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.227015 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 16:54:20.227331 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.227318 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 16:54:20.227379 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.227342 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 16:54:20.229429 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.229411 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 16:54:20.231242 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.231228 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 16:54:20.231315 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.231259 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 16:54:20.231433 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.231423 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 16:54:20.231469 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.231438 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 16:54:20.231469 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.231444 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 16:54:20.231469 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.231450 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 16:54:20.231469 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.231461 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 16:54:20.231469 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.231471 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 16:54:20.232026 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.232017 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 16:54:20.232062 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.232027 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 16:54:20.232062 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.232035 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 16:54:20.232062 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.232041 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 16:54:20.232062 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.232057 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 16:54:20.232172 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.232065 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 16:54:20.232860 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.232851 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 16:54:20.232860 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.232859 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 16:54:20.236033 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.236002 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-228.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 16:54:20.236254 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.236226 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 16:54:20.236313 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.236281 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 16:54:20.237045 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.237029 2559 server.go:1268] "Started kubelet" Jul 10 16:54:20.237138 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.237103 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 16:54:20.237194 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.237131 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 16:54:20.237244 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.237202 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 16:54:20.237987 ip-10-0-141-228 systemd[1]: Started Kubernetes Kubelet. Jul 10 16:54:20.238448 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.238348 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 16:54:20.239071 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.239041 2559 server.go:310] "Adding debug handlers to kubelet server" Jul 10 16:54:20.240242 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.240225 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d954j" Jul 10 16:54:20.242717 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.242694 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 16:54:20.243411 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.241647 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-228.ec2.internal.18c0fbefe76a7620 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-228.ec2.internal,UID:ip-10-0-141-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-228.ec2.internal,},FirstTimestamp:2026-07-10 16:54:20.236985888 +0000 UTC m=+0.362694289,LastTimestamp:2026-07-10 16:54:20.236985888 +0000 UTC m=+0.362694289,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-228.ec2.internal,}" Jul 10 16:54:20.244386 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.244365 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 16:54:20.244386 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.244371 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 16:54:20.244564 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.244477 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:20.245060 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.245043 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 16:54:20.245168 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.245046 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 16:54:20.245274 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.245173 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 16:54:20.245274 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.245236 2559 reconstruct.go:97] "Volume reconstruction finished" Jul 10 16:54:20.245274 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.245246 2559 reconciler.go:29] "Reconciler: start to sync state" Jul 10 16:54:20.245274 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.245252 2559 factory.go:55] Registering systemd factory Jul 10 16:54:20.245455 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.245292 2559 factory.go:223] Registration of the systemd container factory successfully Jul 10 16:54:20.245531 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.245514 2559 factory.go:153] Registering CRI-O factory Jul 10 16:54:20.245531 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.245523 2559 factory.go:223] Registration of the crio container factory successfully Jul 10 16:54:20.245662 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.245568 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 16:54:20.245662 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.245589 2559 factory.go:103] Registering Raw factory Jul 10 16:54:20.245662 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.245600 2559 manager.go:1196] Started watching for new ooms in manager Jul 10 16:54:20.245943 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.245931 2559 manager.go:349] Starting recovery of all containers Jul 10 16:54:20.246353 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.246337 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d954j" Jul 10 16:54:20.247607 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.247581 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 16:54:20.247697 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.247678 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 16:54:20.257862 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.257665 2559 manager.go:354] Recovery completed Jul 10 16:54:20.263994 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.263981 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:20.267000 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.266791 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:20.267000 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.266828 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:20.267000 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.266842 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:20.267937 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.267921 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 16:54:20.267937 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.267934 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:20.268044 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.267936 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 16:54:20.268044 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.267963 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 10 16:54:20.268612 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.268592 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:20.268684 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.268621 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:20.268684 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.268631 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:20.270139 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.270124 2559 policy_none.go:49] "None policy: Start" Jul 10 16:54:20.270190 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.270148 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 16:54:20.270190 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.270159 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 16:54:20.272053 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.272042 2559 policy_none.go:47] "Start" Jul 10 16:54:20.323235 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.323210 2559 manager.go:337] "Starting Device Plugin manager" Jul 10 16:54:20.323374 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.323252 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 16:54:20.323374 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.323262 2559 server.go:87] "Starting device plugin registration server" Jul 10 16:54:20.323607 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.323591 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 16:54:20.323668 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.323611 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 16:54:20.323734 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.323693 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 16:54:20.323783 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.323769 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 16:54:20.323783 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.323777 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 16:54:20.324409 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.324384 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 16:54:20.324536 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.324447 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:20.390555 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.390512 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 16:54:20.392647 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.392040 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 16:54:20.392647 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.392083 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 16:54:20.392647 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.392122 2559 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 16:54:20.392647 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.392166 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 16:54:20.394464 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.394443 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:54:20.424532 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.424429 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:20.425744 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.425730 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:20.425805 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.425759 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:20.425805 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.425769 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:20.425805 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.425791 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.433732 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.433714 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.433793 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.433741 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-228.ec2.internal\": node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:20.448360 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.448338 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:20.493158 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.493111 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-228.ec2.internal"] Jul 10 16:54:20.493239 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.493200 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:20.494888 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.494872 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:20.494940 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.494905 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:20.494940 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.494915 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:20.497201 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.497188 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:20.497301 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.497287 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.497335 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.497318 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:20.498230 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.498215 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:20.498298 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.498250 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:20.498298 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.498216 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:20.498298 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.498263 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:20.498298 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.498276 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:20.498298 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.498285 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:20.500399 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.500384 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.500520 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.500410 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:54:20.501380 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.501365 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:54:20.501439 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.501398 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:54:20.501439 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.501411 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:54:20.524327 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.524299 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-228.ec2.internal\" not found" node="ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.528958 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.528941 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-228.ec2.internal\" not found" node="ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.546942 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.546915 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3a5470e748fa4893099e28e4ea31d34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-228.ec2.internal\" (UID: \"b3a5470e748fa4893099e28e4ea31d34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.547069 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.546947 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7834fe1e11a9e3f6ca100a841c226e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal\" (UID: \"b7834fe1e11a9e3f6ca100a841c226e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.547069 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.546965 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7834fe1e11a9e3f6ca100a841c226e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal\" (UID: \"b7834fe1e11a9e3f6ca100a841c226e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.549023 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.549004 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:20.647711 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.647683 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3a5470e748fa4893099e28e4ea31d34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-228.ec2.internal\" (UID: \"b3a5470e748fa4893099e28e4ea31d34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.647822 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.647722 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3a5470e748fa4893099e28e4ea31d34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-228.ec2.internal\" (UID: \"b3a5470e748fa4893099e28e4ea31d34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.647822 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.647770 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7834fe1e11a9e3f6ca100a841c226e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal\" (UID: \"b7834fe1e11a9e3f6ca100a841c226e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.647822 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.647787 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7834fe1e11a9e3f6ca100a841c226e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal\" (UID: \"b7834fe1e11a9e3f6ca100a841c226e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.647822 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.647812 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7834fe1e11a9e3f6ca100a841c226e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal\" (UID: \"b7834fe1e11a9e3f6ca100a841c226e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.647940 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.647834 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7834fe1e11a9e3f6ca100a841c226e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal\" (UID: \"b7834fe1e11a9e3f6ca100a841c226e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.649751 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.649733 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:20.750613 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.750535 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:20.826776 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.826746 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.831764 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:20.831473 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-228.ec2.internal" Jul 10 16:54:20.851356 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.851327 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:20.951888 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:20.951840 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:21.052589 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:21.052554 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:21.116482 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.116455 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:54:21.153091 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:21.153063 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:21.154085 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.154068 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:54:21.185015 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.184988 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 16:54:21.185602 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.185146 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 16:54:21.185602 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.185188 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 16:54:21.185602 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.185197 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 16:54:21.250194 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.250148 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 16:49:20 +0000 UTC" deadline="2028-02-03 08:41:30.709552365 +0000 UTC" Jul 10 16:54:21.250194 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.250189 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13743h47m9.459367455s" Jul 10 16:54:21.253334 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:21.253297 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:21.353692 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:21.353653 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:21.399656 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:21.399613 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7834fe1e11a9e3f6ca100a841c226e4.slice/crio-8c2a8a5cc73b0b727dc1229c9819e2fa202b3dcbfc71d8d0b7e324988658aa34 WatchSource:0}: Error finding container 8c2a8a5cc73b0b727dc1229c9819e2fa202b3dcbfc71d8d0b7e324988658aa34: Status 404 returned error can't find the container with id 8c2a8a5cc73b0b727dc1229c9819e2fa202b3dcbfc71d8d0b7e324988658aa34 Jul 10 16:54:21.402911 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:21.402882 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a5470e748fa4893099e28e4ea31d34.slice/crio-e46fc0acb6e60f1714b24fc3fa2a670e2cd14482d20ed6059824f4ad249ac29f WatchSource:0}: Error finding container e46fc0acb6e60f1714b24fc3fa2a670e2cd14482d20ed6059824f4ad249ac29f: Status 404 returned error can't find the container with id e46fc0acb6e60f1714b24fc3fa2a670e2cd14482d20ed6059824f4ad249ac29f Jul 10 16:54:21.404816 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.404792 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:54:21.454198 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:21.454154 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:21.555365 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:21.555297 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:21.655589 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:21.655553 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-228.ec2.internal\" not found" Jul 10 16:54:21.754359 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.754328 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:54:21.845146 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.845050 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal" Jul 10 16:54:21.858686 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.858657 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 16:54:21.859867 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.859840 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-228.ec2.internal" Jul 10 16:54:21.870266 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:21.870151 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 16:54:22.228167 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.228072 2559 apiserver.go:52] "Watching apiserver" Jul 10 16:54:22.238423 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.238394 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 16:54:22.239949 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.239914 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-lr5sl","kube-system/konnectivity-agent-4tdzm","kube-system/kube-apiserver-proxy-ip-10-0-141-228.ec2.internal","openshift-dns/node-resolver-vxt8w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal","openshift-multus/multus-additional-cni-plugins-xxchh","openshift-multus/multus-q97mw","openshift-multus/network-metrics-daemon-wg27k","openshift-network-operator/iptables-alerter-78lb5","kube-system/global-pull-secret-syncer-xdv2b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7","openshift-cluster-node-tuning-operator/tuned-grg9k","openshift-image-registry/node-ca-bsmqb","openshift-network-diagnostics/network-check-target-vr2l7"] Jul 10 16:54:22.242689 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.242642 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:22.242822 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.242723 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:22.245076 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.245023 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 16:54:22.245076 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.245038 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4tdzm" Jul 10 16:54:22.247431 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.247390 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:22.247610 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.247469 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:22.247785 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.247759 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 16:54:22.248047 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.248022 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 16:54:22.248194 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.248181 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-chnvg\"" Jul 10 16:54:22.250147 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.249852 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vxt8w" Jul 10 16:54:22.252070 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.252047 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 16:54:22.252740 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.252458 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-94qtp\"" Jul 10 16:54:22.252740 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.252691 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 16:54:22.252935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.252907 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.255591 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.254798 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.255591 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.255553 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 16:54:22.255842 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.255821 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 16:54:22.256192 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.256159 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-stnzf\"" Jul 10 16:54:22.256437 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.256418 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 16:54:22.256707 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.256683 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs\") pod \"network-metrics-daemon-wg27k\" (UID: \"2e1b9d64-bc6b-4a12-86da-7cb47800704b\") " pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:22.256821 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.256720 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ggz9\" (UniqueName: \"kubernetes.io/projected/2e1b9d64-bc6b-4a12-86da-7cb47800704b-kube-api-access-2ggz9\") pod \"network-metrics-daemon-wg27k\" (UID: \"2e1b9d64-bc6b-4a12-86da-7cb47800704b\") " pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:22.256821 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.256748 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/75c5040f-8551-4491-8725-e0b4e51a058a-agent-certs\") pod \"konnectivity-agent-4tdzm\" (UID: \"75c5040f-8551-4491-8725-e0b4e51a058a\") " pod="kube-system/konnectivity-agent-4tdzm" Jul 10 16:54:22.256821 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.256774 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/75c5040f-8551-4491-8725-e0b4e51a058a-konnectivity-ca\") pod \"konnectivity-agent-4tdzm\" (UID: \"75c5040f-8551-4491-8725-e0b4e51a058a\") " pod="kube-system/konnectivity-agent-4tdzm" Jul 10 16:54:22.256821 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.256802 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqw64\" (UniqueName: \"kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64\") pod \"network-check-target-vr2l7\" (UID: \"afc23e34-857f-4c29-a381-cf87340044e7\") " pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:22.257109 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.257089 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 16:54:22.257941 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.257350 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-78lb5" Jul 10 16:54:22.257941 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.257586 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzq62\"" Jul 10 16:54:22.257941 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.257615 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 16:54:22.258107 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.257971 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 16:54:22.258409 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.258387 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 16:54:22.258483 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.258427 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 16:54:22.258702 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.258683 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 16:54:22.259785 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.259762 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:54:22.260861 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.260828 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 16:54:22.260963 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.260891 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.261123 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.261105 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6jdbp\"" Jul 10 16:54:22.262305 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.262285 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 16:54:22.263322 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.263301 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 16:54:22.265123 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.265102 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f8qx5\"" Jul 10 16:54:22.266510 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.266493 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.268985 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.268960 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 16:54:22.269327 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.269309 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jdbq9\"" Jul 10 16:54:22.269560 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.269544 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:22.269781 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.269741 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:22.269781 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.269767 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 16:54:22.269935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.269785 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 16:54:22.269935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.269813 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 16:54:22.269935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.269836 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 16:54:22.269935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.269767 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 16:54:22.272377 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.272363 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.274968 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.274495 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bsmqb" Jul 10 16:54:22.274968 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.274815 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:54:22.275215 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.275193 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4fz8\"" Jul 10 16:54:22.275300 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.275287 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 16:54:22.277068 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.276895 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 16:54:22.277068 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.277044 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 16:54:22.278106 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.277167 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vrzvb\"" Jul 10 16:54:22.278106 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.277198 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 16:54:22.288816 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.288785 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ktfns" Jul 10 16:54:22.298230 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.298199 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ktfns" Jul 10 16:54:22.346189 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.346163 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 16:54:22.357787 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.357757 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-sysconfig\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.358010 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.357792 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7hc7\" (UniqueName: \"kubernetes.io/projected/a63b13f7-1bec-4127-a3ac-9aa89b83b424-kube-api-access-c7hc7\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.358010 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.357812 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db24c\" (UniqueName: \"kubernetes.io/projected/e53d41a0-f680-4189-a2d1-0500d81ca3a9-kube-api-access-db24c\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.358010 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.357835 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-run-ovn\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.358010 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.357854 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-run-ovn-kubernetes\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.358010 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.357870 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd754214-d46d-404b-8d29-699dcd61dd33-env-overrides\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.358010 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.357891 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw64\" (UniqueName: \"kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64\") pod \"network-check-target-vr2l7\" (UID: \"afc23e34-857f-4c29-a381-cf87340044e7\") " pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:22.358010 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.357911 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1cf9f299-96ec-4148-bd09-9641e7d859f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.358010 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.357931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-os-release\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.358010 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.357947 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-run-k8s-cni-cncf-io\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.358413 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358064 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-hostroot\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.358413 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358112 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-node-log\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.358413 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358145 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1cf9f299-96ec-4148-bd09-9641e7d859f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.358413 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358176 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-etc-kubernetes\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.358413 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358205 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7c2a0f7-7fe0-4084-9b87-6c1e95422355-tmp-dir\") pod \"node-resolver-vxt8w\" (UID: \"b7c2a0f7-7fe0-4084-9b87-6c1e95422355\") " pod="openshift-dns/node-resolver-vxt8w" Jul 10 16:54:22.358413 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358230 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-device-dir\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.358413 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358264 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn57x\" (UniqueName: \"kubernetes.io/projected/5dc130ed-3574-4a74-aa2c-e20959046b96-kube-api-access-gn57x\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.358413 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358307 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:22.358413 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358336 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-var-lib-kubelet\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.358413 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358364 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-socket-dir\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.358413 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358394 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-kubelet-config\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:22.358413 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-dbus\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358430 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-host\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358467 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1cf9f299-96ec-4148-bd09-9641e7d859f1-os-release\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358494 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-log-socket\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-cni-netd\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358593 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-sysctl-conf\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358640 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-sys\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358664 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-var-lib-kubelet\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358687 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a63b13f7-1bec-4127-a3ac-9aa89b83b424-tmp\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs\") pod \"network-metrics-daemon-wg27k\" (UID: \"2e1b9d64-bc6b-4a12-86da-7cb47800704b\") " pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358736 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-multus-socket-dir-parent\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358763 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-run-netns\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358784 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-etc-openvswitch\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358802 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4346a8a8-2c9c-4d06-bb87-a5364485b29d-host\") pod \"node-ca-bsmqb\" (UID: \"4346a8a8-2c9c-4d06-bb87-a5364485b29d\") " pod="openshift-image-registry/node-ca-bsmqb" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358837 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.358843 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-sysctl-d\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.359018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358882 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1cf9f299-96ec-4148-bd09-9641e7d859f1-system-cni-dir\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.358898 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1cf9f299-96ec-4148-bd09-9641e7d859f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.358941 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs podName:2e1b9d64-bc6b-4a12-86da-7cb47800704b nodeName:}" failed. No retries permitted until 2026-07-10 16:54:22.858901086 +0000 UTC m=+2.984609499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs") pod "network-metrics-daemon-wg27k" (UID: "2e1b9d64-bc6b-4a12-86da-7cb47800704b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359055 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb371128-ff07-47e7-95a3-9815145b3724-host-slash\") pod \"iptables-alerter-78lb5\" (UID: \"cb371128-ff07-47e7-95a3-9815145b3724\") " pod="openshift-network-operator/iptables-alerter-78lb5" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359076 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-multus-cni-dir\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359100 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-kubelet\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359117 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd754214-d46d-404b-8d29-699dcd61dd33-ovnkube-script-lib\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359146 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-sys-fs\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359161 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-systemd\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359201 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/75c5040f-8551-4491-8725-e0b4e51a058a-konnectivity-ca\") pod \"konnectivity-agent-4tdzm\" (UID: \"75c5040f-8551-4491-8725-e0b4e51a058a\") " pod="kube-system/konnectivity-agent-4tdzm" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359229 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1cf9f299-96ec-4148-bd09-9641e7d859f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359248 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqrq\" (UniqueName: \"kubernetes.io/projected/1cf9f299-96ec-4148-bd09-9641e7d859f1-kube-api-access-tfqrq\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359266 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-multus-conf-dir\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359283 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7c2a0f7-7fe0-4084-9b87-6c1e95422355-hosts-file\") pod \"node-resolver-vxt8w\" (UID: \"b7c2a0f7-7fe0-4084-9b87-6c1e95422355\") " pod="openshift-dns/node-resolver-vxt8w" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ggz9\" (UniqueName: \"kubernetes.io/projected/2e1b9d64-bc6b-4a12-86da-7cb47800704b-kube-api-access-2ggz9\") pod \"network-metrics-daemon-wg27k\" (UID: \"2e1b9d64-bc6b-4a12-86da-7cb47800704b\") " pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359687 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/75c5040f-8551-4491-8725-e0b4e51a058a-agent-certs\") pod \"konnectivity-agent-4tdzm\" (UID: \"75c5040f-8551-4491-8725-e0b4e51a058a\") " pod="kube-system/konnectivity-agent-4tdzm" Jul 10 16:54:22.359900 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359704 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-system-cni-dir\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359721 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e53d41a0-f680-4189-a2d1-0500d81ca3a9-multus-daemon-config\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359736 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-run-systemd\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359753 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-cni-bin\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359796 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxgng\" (UniqueName: \"kubernetes.io/projected/dd754214-d46d-404b-8d29-699dcd61dd33-kube-api-access-kxgng\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359839 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-var-lib-cni-bin\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359867 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/75c5040f-8551-4491-8725-e0b4e51a058a-konnectivity-ca\") pod \"konnectivity-agent-4tdzm\" (UID: \"75c5040f-8551-4491-8725-e0b4e51a058a\") " pod="kube-system/konnectivity-agent-4tdzm" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-run-multus-certs\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359914 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-run-netns\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359951 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-modprobe-d\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.359977 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1cf9f299-96ec-4148-bd09-9641e7d859f1-cnibin\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360005 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-systemd-units\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360048 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-slash\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360078 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd754214-d46d-404b-8d29-699dcd61dd33-ovn-node-metrics-cert\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360086 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360111 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d6gz\" (UniqueName: \"kubernetes.io/projected/b7c2a0f7-7fe0-4084-9b87-6c1e95422355-kube-api-access-5d6gz\") pod \"node-resolver-vxt8w\" (UID: \"b7c2a0f7-7fe0-4084-9b87-6c1e95422355\") " pod="openshift-dns/node-resolver-vxt8w" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360137 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-run\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.360546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360164 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-tuned\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360191 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4pp\" (UniqueName: \"kubernetes.io/projected/4346a8a8-2c9c-4d06-bb87-a5364485b29d-kube-api-access-km4pp\") pod \"node-ca-bsmqb\" (UID: \"4346a8a8-2c9c-4d06-bb87-a5364485b29d\") " pod="openshift-image-registry/node-ca-bsmqb" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360231 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-registration-dir\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360270 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-kubernetes\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360292 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-lib-modules\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360309 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360330 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdwz5\" (UniqueName: \"kubernetes.io/projected/cb371128-ff07-47e7-95a3-9815145b3724-kube-api-access-jdwz5\") pod \"iptables-alerter-78lb5\" (UID: \"cb371128-ff07-47e7-95a3-9815145b3724\") " pod="openshift-network-operator/iptables-alerter-78lb5" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360358 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-cnibin\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360378 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e53d41a0-f680-4189-a2d1-0500d81ca3a9-cni-binary-copy\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360394 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-var-lib-cni-multus\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360409 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360430 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4346a8a8-2c9c-4d06-bb87-a5364485b29d-serviceca\") pod \"node-ca-bsmqb\" (UID: \"4346a8a8-2c9c-4d06-bb87-a5364485b29d\") " pod="openshift-image-registry/node-ca-bsmqb" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360454 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb371128-ff07-47e7-95a3-9815145b3724-iptables-alerter-script\") pod \"iptables-alerter-78lb5\" (UID: \"cb371128-ff07-47e7-95a3-9815145b3724\") " pod="openshift-network-operator/iptables-alerter-78lb5" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360475 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-var-lib-openvswitch\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360491 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-run-openvswitch\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.361118 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.360529 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd754214-d46d-404b-8d29-699dcd61dd33-ovnkube-config\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.363658 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.363636 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/75c5040f-8551-4491-8725-e0b4e51a058a-agent-certs\") pod \"konnectivity-agent-4tdzm\" (UID: \"75c5040f-8551-4491-8725-e0b4e51a058a\") " pod="kube-system/konnectivity-agent-4tdzm" Jul 10 16:54:22.368769 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.368745 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:54:22.368769 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.368771 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:54:22.368940 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.368787 2559 projected.go:196] Error preparing data for projected volume kube-api-access-lqw64 for pod openshift-network-diagnostics/network-check-target-vr2l7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:22.368940 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.368872 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64 podName:afc23e34-857f-4c29-a381-cf87340044e7 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:22.868852546 +0000 UTC m=+2.994560956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lqw64" (UniqueName: "kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64") pod "network-check-target-vr2l7" (UID: "afc23e34-857f-4c29-a381-cf87340044e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:22.372122 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.372099 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ggz9\" (UniqueName: \"kubernetes.io/projected/2e1b9d64-bc6b-4a12-86da-7cb47800704b-kube-api-access-2ggz9\") pod \"network-metrics-daemon-wg27k\" (UID: \"2e1b9d64-bc6b-4a12-86da-7cb47800704b\") " pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:22.400538 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.400458 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-228.ec2.internal" event={"ID":"b3a5470e748fa4893099e28e4ea31d34","Type":"ContainerStarted","Data":"e46fc0acb6e60f1714b24fc3fa2a670e2cd14482d20ed6059824f4ad249ac29f"} Jul 10 16:54:22.404933 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.404895 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal" event={"ID":"b7834fe1e11a9e3f6ca100a841c226e4","Type":"ContainerStarted","Data":"8c2a8a5cc73b0b727dc1229c9819e2fa202b3dcbfc71d8d0b7e324988658aa34"} Jul 10 16:54:22.461679 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461635 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-multus-socket-dir-parent\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.461872 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461686 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-run-netns\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.461872 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461717 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-etc-openvswitch\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.461872 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461742 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4346a8a8-2c9c-4d06-bb87-a5364485b29d-host\") pod \"node-ca-bsmqb\" (UID: \"4346a8a8-2c9c-4d06-bb87-a5364485b29d\") " pod="openshift-image-registry/node-ca-bsmqb" Jul 10 16:54:22.461872 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461771 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.461872 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461798 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-sysctl-d\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.461872 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461793 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-multus-socket-dir-parent\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.461872 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461826 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1cf9f299-96ec-4148-bd09-9641e7d859f1-system-cni-dir\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.461872 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461833 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4346a8a8-2c9c-4d06-bb87-a5364485b29d-host\") pod \"node-ca-bsmqb\" (UID: \"4346a8a8-2c9c-4d06-bb87-a5364485b29d\") " pod="openshift-image-registry/node-ca-bsmqb" Jul 10 16:54:22.461872 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1cf9f299-96ec-4148-bd09-9641e7d859f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.461872 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461872 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461883 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb371128-ff07-47e7-95a3-9815145b3724-host-slash\") pod \"iptables-alerter-78lb5\" (UID: \"cb371128-ff07-47e7-95a3-9815145b3724\") " pod="openshift-network-operator/iptables-alerter-78lb5" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461892 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-etc-openvswitch\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-multus-cni-dir\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461928 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1cf9f299-96ec-4148-bd09-9641e7d859f1-system-cni-dir\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461933 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-kubelet\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461965 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd754214-d46d-404b-8d29-699dcd61dd33-ovnkube-script-lib\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461985 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-kubelet\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.461993 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-sys-fs\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462016 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-systemd\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462024 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-sysctl-d\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462034 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1cf9f299-96ec-4148-bd09-9641e7d859f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462058 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqrq\" (UniqueName: \"kubernetes.io/projected/1cf9f299-96ec-4148-bd09-9641e7d859f1-kube-api-access-tfqrq\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462077 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-sys-fs\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462088 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-multus-conf-dir\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7c2a0f7-7fe0-4084-9b87-6c1e95422355-hosts-file\") pod \"node-resolver-vxt8w\" (UID: \"b7c2a0f7-7fe0-4084-9b87-6c1e95422355\") " pod="openshift-dns/node-resolver-vxt8w" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-system-cni-dir\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.462311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462185 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e53d41a0-f680-4189-a2d1-0500d81ca3a9-multus-daemon-config\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462206 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-run-systemd\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462233 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-cni-bin\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462258 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxgng\" (UniqueName: \"kubernetes.io/projected/dd754214-d46d-404b-8d29-699dcd61dd33-kube-api-access-kxgng\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462291 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-var-lib-cni-bin\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-run-multus-certs\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462350 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-run-netns\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462374 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-modprobe-d\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462404 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1cf9f299-96ec-4148-bd09-9641e7d859f1-cnibin\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462435 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-systemd-units\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462462 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-slash\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462499 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd754214-d46d-404b-8d29-699dcd61dd33-ovn-node-metrics-cert\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462525 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d6gz\" (UniqueName: \"kubernetes.io/projected/b7c2a0f7-7fe0-4084-9b87-6c1e95422355-kube-api-access-5d6gz\") pod \"node-resolver-vxt8w\" (UID: \"b7c2a0f7-7fe0-4084-9b87-6c1e95422355\") " pod="openshift-dns/node-resolver-vxt8w" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462545 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-run\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462608 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-tuned\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4pp\" (UniqueName: \"kubernetes.io/projected/4346a8a8-2c9c-4d06-bb87-a5364485b29d-kube-api-access-km4pp\") pod \"node-ca-bsmqb\" (UID: \"4346a8a8-2c9c-4d06-bb87-a5364485b29d\") " pod="openshift-image-registry/node-ca-bsmqb" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-registration-dir\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.463059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462659 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1cf9f299-96ec-4148-bd09-9641e7d859f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462678 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-kubernetes\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462715 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-lib-modules\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462721 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-kubernetes\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462745 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462769 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-var-lib-cni-bin\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462775 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdwz5\" (UniqueName: \"kubernetes.io/projected/cb371128-ff07-47e7-95a3-9815145b3724-kube-api-access-jdwz5\") pod \"iptables-alerter-78lb5\" (UID: \"cb371128-ff07-47e7-95a3-9815145b3724\") " pod="openshift-network-operator/iptables-alerter-78lb5" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462816 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-cnibin\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462838 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb371128-ff07-47e7-95a3-9815145b3724-host-slash\") pod \"iptables-alerter-78lb5\" (UID: \"cb371128-ff07-47e7-95a3-9815145b3724\") " pod="openshift-network-operator/iptables-alerter-78lb5" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462849 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e53d41a0-f680-4189-a2d1-0500d81ca3a9-cni-binary-copy\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462875 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-var-lib-cni-multus\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462893 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462914 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4346a8a8-2c9c-4d06-bb87-a5364485b29d-serviceca\") pod \"node-ca-bsmqb\" (UID: \"4346a8a8-2c9c-4d06-bb87-a5364485b29d\") " pod="openshift-image-registry/node-ca-bsmqb" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462922 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-multus-cni-dir\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462931 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb371128-ff07-47e7-95a3-9815145b3724-iptables-alerter-script\") pod \"iptables-alerter-78lb5\" (UID: \"cb371128-ff07-47e7-95a3-9815145b3724\") " pod="openshift-network-operator/iptables-alerter-78lb5" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462934 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-systemd\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462954 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-var-lib-openvswitch\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.463887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462971 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-run-openvswitch\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.462997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd754214-d46d-404b-8d29-699dcd61dd33-ovnkube-config\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463025 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-sysconfig\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463039 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463058 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hc7\" (UniqueName: \"kubernetes.io/projected/a63b13f7-1bec-4127-a3ac-9aa89b83b424-kube-api-access-c7hc7\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463091 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db24c\" (UniqueName: \"kubernetes.io/projected/e53d41a0-f680-4189-a2d1-0500d81ca3a9-kube-api-access-db24c\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463101 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1cf9f299-96ec-4148-bd09-9641e7d859f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463116 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-run-ovn\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463139 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-run-ovn-kubernetes\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd754214-d46d-404b-8d29-699dcd61dd33-env-overrides\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463175 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1cf9f299-96ec-4148-bd09-9641e7d859f1-cnibin\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463215 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-modprobe-d\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463233 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-systemd-units\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463256 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-multus-conf-dir\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463278 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-run-netns\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463321 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-slash\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7c2a0f7-7fe0-4084-9b87-6c1e95422355-hosts-file\") pod \"node-resolver-vxt8w\" (UID: \"b7c2a0f7-7fe0-4084-9b87-6c1e95422355\") " pod="openshift-dns/node-resolver-vxt8w" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463530 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-system-cni-dir\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.464736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463572 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-run-openvswitch\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463654 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-cnibin\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463707 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-var-lib-openvswitch\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463728 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-lib-modules\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463765 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-cni-bin\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463813 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-run-systemd\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463866 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-var-lib-cni-multus\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463931 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-run\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464056 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd754214-d46d-404b-8d29-699dcd61dd33-ovnkube-config\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464134 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-sysconfig\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464209 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464262 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e53d41a0-f680-4189-a2d1-0500d81ca3a9-cni-binary-copy\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb371128-ff07-47e7-95a3-9815145b3724-iptables-alerter-script\") pod \"iptables-alerter-78lb5\" (UID: \"cb371128-ff07-47e7-95a3-9815145b3724\") " pod="openshift-network-operator/iptables-alerter-78lb5" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464328 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e53d41a0-f680-4189-a2d1-0500d81ca3a9-multus-daemon-config\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-run-ovn-kubernetes\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464409 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-run-ovn\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.463103 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-registration-dir\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464455 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-run-multus-certs\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.465638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464469 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1cf9f299-96ec-4148-bd09-9641e7d859f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464502 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-os-release\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464533 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-run-k8s-cni-cncf-io\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464588 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-hostroot\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464622 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-node-log\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464679 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-node-log\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464689 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4346a8a8-2c9c-4d06-bb87-a5364485b29d-serviceca\") pod \"node-ca-bsmqb\" (UID: \"4346a8a8-2c9c-4d06-bb87-a5364485b29d\") " pod="openshift-image-registry/node-ca-bsmqb" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464694 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-os-release\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464735 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-hostroot\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464742 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-run-k8s-cni-cncf-io\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464765 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1cf9f299-96ec-4148-bd09-9641e7d859f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464794 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-etc-kubernetes\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464812 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7c2a0f7-7fe0-4084-9b87-6c1e95422355-tmp-dir\") pod \"node-resolver-vxt8w\" (UID: \"b7c2a0f7-7fe0-4084-9b87-6c1e95422355\") " pod="openshift-dns/node-resolver-vxt8w" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464835 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-device-dir\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464861 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn57x\" (UniqueName: \"kubernetes.io/projected/5dc130ed-3574-4a74-aa2c-e20959046b96-kube-api-access-gn57x\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464891 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464924 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-var-lib-kubelet\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464948 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd754214-d46d-404b-8d29-699dcd61dd33-env-overrides\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.466474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1cf9f299-96ec-4148-bd09-9641e7d859f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465032 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-device-dir\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465086 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-var-lib-kubelet\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.465112 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.465175 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret podName:2a54dfc8-2f7b-4e80-9f23-03c128feeb24 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:22.96515569 +0000 UTC m=+3.090864082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret") pod "global-pull-secret-syncer-xdv2b" (UID: "2a54dfc8-2f7b-4e80-9f23-03c128feeb24") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465268 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1cf9f299-96ec-4148-bd09-9641e7d859f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.464954 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-socket-dir\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465332 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-kubelet-config\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465360 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-dbus\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465371 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5dc130ed-3574-4a74-aa2c-e20959046b96-socket-dir\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465386 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-host\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-etc-kubernetes\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465410 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7c2a0f7-7fe0-4084-9b87-6c1e95422355-tmp-dir\") pod \"node-resolver-vxt8w\" (UID: \"b7c2a0f7-7fe0-4084-9b87-6c1e95422355\") " pod="openshift-dns/node-resolver-vxt8w" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465433 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-kubelet-config\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-host\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465488 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-dbus\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465499 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1cf9f299-96ec-4148-bd09-9641e7d859f1-os-release\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.467195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465415 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1cf9f299-96ec-4148-bd09-9641e7d859f1-os-release\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465536 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-log-socket\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465589 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-cni-netd\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465617 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-sysctl-conf\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465639 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-sys\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-var-lib-kubelet\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd754214-d46d-404b-8d29-699dcd61dd33-ovnkube-script-lib\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465685 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a63b13f7-1bec-4127-a3ac-9aa89b83b424-tmp\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465688 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-log-socket\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465755 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-var-lib-kubelet\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465810 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd754214-d46d-404b-8d29-699dcd61dd33-host-cni-netd\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465844 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-sysctl-conf\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.465878 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e53d41a0-f680-4189-a2d1-0500d81ca3a9-host-run-netns\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.466034 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a63b13f7-1bec-4127-a3ac-9aa89b83b424-sys\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.466480 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd754214-d46d-404b-8d29-699dcd61dd33-ovn-node-metrics-cert\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.467954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.467327 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a63b13f7-1bec-4127-a3ac-9aa89b83b424-etc-tuned\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.468492 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.468200 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a63b13f7-1bec-4127-a3ac-9aa89b83b424-tmp\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.475135 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.475106 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxgng\" (UniqueName: \"kubernetes.io/projected/dd754214-d46d-404b-8d29-699dcd61dd33-kube-api-access-kxgng\") pod \"ovnkube-node-lr5sl\" (UID: \"dd754214-d46d-404b-8d29-699dcd61dd33\") " pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.479597 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.477762 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d6gz\" (UniqueName: \"kubernetes.io/projected/b7c2a0f7-7fe0-4084-9b87-6c1e95422355-kube-api-access-5d6gz\") pod \"node-resolver-vxt8w\" (UID: \"b7c2a0f7-7fe0-4084-9b87-6c1e95422355\") " pod="openshift-dns/node-resolver-vxt8w" Jul 10 16:54:22.479597 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.478031 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4pp\" (UniqueName: \"kubernetes.io/projected/4346a8a8-2c9c-4d06-bb87-a5364485b29d-kube-api-access-km4pp\") pod \"node-ca-bsmqb\" (UID: \"4346a8a8-2c9c-4d06-bb87-a5364485b29d\") " pod="openshift-image-registry/node-ca-bsmqb" Jul 10 16:54:22.479597 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.478504 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdwz5\" (UniqueName: \"kubernetes.io/projected/cb371128-ff07-47e7-95a3-9815145b3724-kube-api-access-jdwz5\") pod \"iptables-alerter-78lb5\" (UID: \"cb371128-ff07-47e7-95a3-9815145b3724\") " pod="openshift-network-operator/iptables-alerter-78lb5" Jul 10 16:54:22.479597 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.478771 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqrq\" (UniqueName: \"kubernetes.io/projected/1cf9f299-96ec-4148-bd09-9641e7d859f1-kube-api-access-tfqrq\") pod \"multus-additional-cni-plugins-xxchh\" (UID: \"1cf9f299-96ec-4148-bd09-9641e7d859f1\") " pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.479597 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.478846 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db24c\" (UniqueName: \"kubernetes.io/projected/e53d41a0-f680-4189-a2d1-0500d81ca3a9-kube-api-access-db24c\") pod \"multus-q97mw\" (UID: \"e53d41a0-f680-4189-a2d1-0500d81ca3a9\") " pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.479597 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.478898 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn57x\" (UniqueName: \"kubernetes.io/projected/5dc130ed-3574-4a74-aa2c-e20959046b96-kube-api-access-gn57x\") pod \"aws-ebs-csi-driver-node-bkpr7\" (UID: \"5dc130ed-3574-4a74-aa2c-e20959046b96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.480294 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.480273 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hc7\" (UniqueName: \"kubernetes.io/projected/a63b13f7-1bec-4127-a3ac-9aa89b83b424-kube-api-access-c7hc7\") pod \"tuned-grg9k\" (UID: \"a63b13f7-1bec-4127-a3ac-9aa89b83b424\") " pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.530372 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.530339 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:54:22.559939 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.559893 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4tdzm" Jul 10 16:54:22.572247 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.572221 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vxt8w" Jul 10 16:54:22.582281 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.582258 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" Jul 10 16:54:22.589296 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.589269 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xxchh" Jul 10 16:54:22.600302 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.600281 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-78lb5" Jul 10 16:54:22.608292 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.608267 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q97mw" Jul 10 16:54:22.617424 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.617400 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:22.625661 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.625635 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-grg9k" Jul 10 16:54:22.631623 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.631596 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bsmqb" Jul 10 16:54:22.867644 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.867533 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:54:22.869422 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.869393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw64\" (UniqueName: \"kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64\") pod \"network-check-target-vr2l7\" (UID: \"afc23e34-857f-4c29-a381-cf87340044e7\") " pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:22.869597 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.869462 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs\") pod \"network-metrics-daemon-wg27k\" (UID: \"2e1b9d64-bc6b-4a12-86da-7cb47800704b\") " pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:22.869640 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.869600 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:22.869640 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.869607 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:54:22.869640 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.869639 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:54:22.869749 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.869655 2559 projected.go:196] Error preparing data for projected volume kube-api-access-lqw64 for pod openshift-network-diagnostics/network-check-target-vr2l7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:22.869749 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.869668 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs podName:2e1b9d64-bc6b-4a12-86da-7cb47800704b nodeName:}" failed. No retries permitted until 2026-07-10 16:54:23.869649107 +0000 UTC m=+3.995357502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs") pod "network-metrics-daemon-wg27k" (UID: "2e1b9d64-bc6b-4a12-86da-7cb47800704b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:22.869749 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.869701 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64 podName:afc23e34-857f-4c29-a381-cf87340044e7 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:23.869688424 +0000 UTC m=+3.995396813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lqw64" (UniqueName: "kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64") pod "network-check-target-vr2l7" (UID: "afc23e34-857f-4c29-a381-cf87340044e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:22.970401 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:22.970361 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:22.970624 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.970574 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:22.970679 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:22.970647 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret podName:2a54dfc8-2f7b-4e80-9f23-03c128feeb24 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:23.970627628 +0000 UTC m=+4.096336036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret") pod "global-pull-secret-syncer-xdv2b" (UID: "2a54dfc8-2f7b-4e80-9f23-03c128feeb24") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:23.302534 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:23.302493 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 16:49:22 +0000 UTC" deadline="2028-01-12 07:24:07.040607536 +0000 UTC" Jul 10 16:54:23.302534 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:23.302542 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13214h29m43.738069348s" Jul 10 16:54:23.392798 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:23.392717 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:23.392962 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:23.392870 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:23.446779 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:23.446638 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf9f299_96ec_4148_bd09_9641e7d859f1.slice/crio-0c11e2fadbaf17b03e05904d5579a4ef537e8470cf4919f9b68476dadf5bcf70 WatchSource:0}: Error finding container 0c11e2fadbaf17b03e05904d5579a4ef537e8470cf4919f9b68476dadf5bcf70: Status 404 returned error can't find the container with id 0c11e2fadbaf17b03e05904d5579a4ef537e8470cf4919f9b68476dadf5bcf70 Jul 10 16:54:23.449931 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:23.449901 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53d41a0_f680_4189_a2d1_0500d81ca3a9.slice/crio-8d80e0fab864c4ea745a8e74053d347073a44510b318cd60f80938fbc853ea3d WatchSource:0}: Error finding container 8d80e0fab864c4ea745a8e74053d347073a44510b318cd60f80938fbc853ea3d: Status 404 returned error can't find the container with id 8d80e0fab864c4ea745a8e74053d347073a44510b318cd60f80938fbc853ea3d Jul 10 16:54:23.458670 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:23.458641 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63b13f7_1bec_4127_a3ac_9aa89b83b424.slice/crio-c46253e1ede89ebacf206f0a0d10164534829a44f378995cf8863fe2d4ea8642 WatchSource:0}: Error finding container c46253e1ede89ebacf206f0a0d10164534829a44f378995cf8863fe2d4ea8642: Status 404 returned error can't find the container with id c46253e1ede89ebacf206f0a0d10164534829a44f378995cf8863fe2d4ea8642 Jul 10 16:54:23.459874 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:23.459838 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c5040f_8551_4491_8725_e0b4e51a058a.slice/crio-afe566e363ef994e60f268cc30bac9612254d275dab695b62b4d93ba6c7af5a5 WatchSource:0}: Error finding container afe566e363ef994e60f268cc30bac9612254d275dab695b62b4d93ba6c7af5a5: Status 404 returned error can't find the container with id afe566e363ef994e60f268cc30bac9612254d275dab695b62b4d93ba6c7af5a5 Jul 10 16:54:23.461684 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:23.461658 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c2a0f7_7fe0_4084_9b87_6c1e95422355.slice/crio-5795d19b998c7a4fb4ae40c92f15fc6732bb00e5283b67a9a5ece9b8102bf1a3 WatchSource:0}: Error finding container 5795d19b998c7a4fb4ae40c92f15fc6732bb00e5283b67a9a5ece9b8102bf1a3: Status 404 returned error can't find the container with id 5795d19b998c7a4fb4ae40c92f15fc6732bb00e5283b67a9a5ece9b8102bf1a3 Jul 10 16:54:23.462740 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:23.462715 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb371128_ff07_47e7_95a3_9815145b3724.slice/crio-1f8e025bb73177b3967a1788e2c29dc86135dbebaa0411ca8f27c0e6ee4aeea7 WatchSource:0}: Error finding container 1f8e025bb73177b3967a1788e2c29dc86135dbebaa0411ca8f27c0e6ee4aeea7: Status 404 returned error can't find the container with id 1f8e025bb73177b3967a1788e2c29dc86135dbebaa0411ca8f27c0e6ee4aeea7 Jul 10 16:54:23.464176 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:23.464151 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc130ed_3574_4a74_aa2c_e20959046b96.slice/crio-bb3700eca99008fc6b797db0aa9c3a70758c0ebb7be3b581d6f451cf69dfb5f9 WatchSource:0}: Error finding container bb3700eca99008fc6b797db0aa9c3a70758c0ebb7be3b581d6f451cf69dfb5f9: Status 404 returned error can't find the container with id bb3700eca99008fc6b797db0aa9c3a70758c0ebb7be3b581d6f451cf69dfb5f9 Jul 10 16:54:23.466840 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:23.466809 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4346a8a8_2c9c_4d06_bb87_a5364485b29d.slice/crio-9a9ab93e7202aeb20ec1ddbfd7cdefd9be18e367def30b0089a88ec0eab540ea WatchSource:0}: Error finding container 9a9ab93e7202aeb20ec1ddbfd7cdefd9be18e367def30b0089a88ec0eab540ea: Status 404 returned error can't find the container with id 9a9ab93e7202aeb20ec1ddbfd7cdefd9be18e367def30b0089a88ec0eab540ea Jul 10 16:54:23.491451 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:23.491425 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd754214_d46d_404b_8d29_699dcd61dd33.slice/crio-a21532e3958bd82197648e9d09a3c2da39b825790440cf09242ad335ab975b36 WatchSource:0}: Error finding container a21532e3958bd82197648e9d09a3c2da39b825790440cf09242ad335ab975b36: Status 404 returned error can't find the container with id a21532e3958bd82197648e9d09a3c2da39b825790440cf09242ad335ab975b36 Jul 10 16:54:23.878111 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:23.877909 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs\") pod \"network-metrics-daemon-wg27k\" (UID: \"2e1b9d64-bc6b-4a12-86da-7cb47800704b\") " pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:23.878292 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:23.878150 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw64\" (UniqueName: \"kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64\") pod \"network-check-target-vr2l7\" (UID: \"afc23e34-857f-4c29-a381-cf87340044e7\") " pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:23.878292 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:23.878072 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:23.878292 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:23.878255 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:54:23.878292 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:23.878268 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:54:23.878292 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:23.878276 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs podName:2e1b9d64-bc6b-4a12-86da-7cb47800704b nodeName:}" failed. No retries permitted until 2026-07-10 16:54:25.878256621 +0000 UTC m=+6.003965015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs") pod "network-metrics-daemon-wg27k" (UID: "2e1b9d64-bc6b-4a12-86da-7cb47800704b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:23.878292 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:23.878279 2559 projected.go:196] Error preparing data for projected volume kube-api-access-lqw64 for pod openshift-network-diagnostics/network-check-target-vr2l7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:23.878511 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:23.878322 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64 podName:afc23e34-857f-4c29-a381-cf87340044e7 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:25.87831036 +0000 UTC m=+6.004018750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lqw64" (UniqueName: "kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64") pod "network-check-target-vr2l7" (UID: "afc23e34-857f-4c29-a381-cf87340044e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:23.979730 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:23.979002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:23.979730 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:23.979205 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:23.979730 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:23.979285 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret podName:2a54dfc8-2f7b-4e80-9f23-03c128feeb24 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:25.979263041 +0000 UTC m=+6.104971450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret") pod "global-pull-secret-syncer-xdv2b" (UID: "2a54dfc8-2f7b-4e80-9f23-03c128feeb24") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:24.018979 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.018948 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:54:24.303234 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.303112 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 16:49:22 +0000 UTC" deadline="2028-01-01 09:51:43.167998836 +0000 UTC" Jul 10 16:54:24.303234 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.303178 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12952h57m18.864825793s" Jul 10 16:54:24.398162 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.398112 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:24.398321 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:24.398246 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:24.398728 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.398703 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:24.398822 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:24.398808 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:24.422908 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.422869 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bsmqb" event={"ID":"4346a8a8-2c9c-4d06-bb87-a5364485b29d","Type":"ContainerStarted","Data":"9a9ab93e7202aeb20ec1ddbfd7cdefd9be18e367def30b0089a88ec0eab540ea"} Jul 10 16:54:24.435594 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.435493 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-78lb5" event={"ID":"cb371128-ff07-47e7-95a3-9815145b3724","Type":"ContainerStarted","Data":"1f8e025bb73177b3967a1788e2c29dc86135dbebaa0411ca8f27c0e6ee4aeea7"} Jul 10 16:54:24.439434 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.439353 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vxt8w" event={"ID":"b7c2a0f7-7fe0-4084-9b87-6c1e95422355","Type":"ContainerStarted","Data":"5795d19b998c7a4fb4ae40c92f15fc6732bb00e5283b67a9a5ece9b8102bf1a3"} Jul 10 16:54:24.444571 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.444450 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-grg9k" event={"ID":"a63b13f7-1bec-4127-a3ac-9aa89b83b424","Type":"ContainerStarted","Data":"c46253e1ede89ebacf206f0a0d10164534829a44f378995cf8863fe2d4ea8642"} Jul 10 16:54:24.453170 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.452913 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxchh" event={"ID":"1cf9f299-96ec-4148-bd09-9641e7d859f1","Type":"ContainerStarted","Data":"0c11e2fadbaf17b03e05904d5579a4ef537e8470cf4919f9b68476dadf5bcf70"} Jul 10 16:54:24.463632 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.462949 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-228.ec2.internal" event={"ID":"b3a5470e748fa4893099e28e4ea31d34","Type":"ContainerStarted","Data":"21f4a7bf19673d844c7c0cdec7a9c2f3c9f09650bbe484d79b4962839a0d7f76"} Jul 10 16:54:24.477467 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.476801 2559 generic.go:356] "Generic (PLEG): container finished" podID="b7834fe1e11a9e3f6ca100a841c226e4" containerID="c60a8d11babd460b6b982fe326b7234a7cf33cd28128facdd98abc73bb83c8e9" exitCode=0 Jul 10 16:54:24.477467 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.476891 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal" event={"ID":"b7834fe1e11a9e3f6ca100a841c226e4","Type":"ContainerDied","Data":"c60a8d11babd460b6b982fe326b7234a7cf33cd28128facdd98abc73bb83c8e9"} Jul 10 16:54:24.491980 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.491944 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" event={"ID":"dd754214-d46d-404b-8d29-699dcd61dd33","Type":"ContainerStarted","Data":"a21532e3958bd82197648e9d09a3c2da39b825790440cf09242ad335ab975b36"} Jul 10 16:54:24.498176 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.498113 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-228.ec2.internal" podStartSLOduration=2.49809015 podStartE2EDuration="2.49809015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:24.480532117 +0000 UTC m=+4.606240531" watchObservedRunningTime="2026-07-10 16:54:24.49809015 +0000 UTC m=+4.623798566" Jul 10 16:54:24.504938 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.504905 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" event={"ID":"5dc130ed-3574-4a74-aa2c-e20959046b96","Type":"ContainerStarted","Data":"bb3700eca99008fc6b797db0aa9c3a70758c0ebb7be3b581d6f451cf69dfb5f9"} Jul 10 16:54:24.511297 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.511264 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4tdzm" event={"ID":"75c5040f-8551-4491-8725-e0b4e51a058a","Type":"ContainerStarted","Data":"afe566e363ef994e60f268cc30bac9612254d275dab695b62b4d93ba6c7af5a5"} Jul 10 16:54:24.521795 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:24.521758 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q97mw" event={"ID":"e53d41a0-f680-4189-a2d1-0500d81ca3a9","Type":"ContainerStarted","Data":"8d80e0fab864c4ea745a8e74053d347073a44510b318cd60f80938fbc853ea3d"} Jul 10 16:54:25.392645 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:25.392602 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:25.393152 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:25.392774 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:25.556221 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:25.556174 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal" event={"ID":"b7834fe1e11a9e3f6ca100a841c226e4","Type":"ContainerStarted","Data":"fc2d9535452c90e7408643f4b567fcfe7cf4c4e697d7e2750fbe9e4f3ab0e6d1"} Jul 10 16:54:25.895119 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:25.895079 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw64\" (UniqueName: \"kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64\") pod \"network-check-target-vr2l7\" (UID: \"afc23e34-857f-4c29-a381-cf87340044e7\") " pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:25.895335 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:25.895154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs\") pod \"network-metrics-daemon-wg27k\" (UID: \"2e1b9d64-bc6b-4a12-86da-7cb47800704b\") " pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:25.895335 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:25.895286 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:25.895335 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:25.895296 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:54:25.895335 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:25.895325 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:54:25.895335 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:25.895341 2559 projected.go:196] Error preparing data for projected volume kube-api-access-lqw64 for pod openshift-network-diagnostics/network-check-target-vr2l7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:25.895629 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:25.895351 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs podName:2e1b9d64-bc6b-4a12-86da-7cb47800704b nodeName:}" failed. No retries permitted until 2026-07-10 16:54:29.89533103 +0000 UTC m=+10.021039425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs") pod "network-metrics-daemon-wg27k" (UID: "2e1b9d64-bc6b-4a12-86da-7cb47800704b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:25.895629 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:25.895400 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64 podName:afc23e34-857f-4c29-a381-cf87340044e7 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:29.895380893 +0000 UTC m=+10.021089317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lqw64" (UniqueName: "kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64") pod "network-check-target-vr2l7" (UID: "afc23e34-857f-4c29-a381-cf87340044e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:25.995591 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:25.995531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:25.995771 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:25.995701 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:25.995827 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:25.995780 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret podName:2a54dfc8-2f7b-4e80-9f23-03c128feeb24 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:29.995757845 +0000 UTC m=+10.121466236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret") pod "global-pull-secret-syncer-xdv2b" (UID: "2a54dfc8-2f7b-4e80-9f23-03c128feeb24") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:26.395896 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:26.393641 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:26.395896 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:26.393659 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:26.395896 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:26.393794 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:26.395896 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:26.393932 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:27.392595 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:27.392567 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:27.392775 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:27.392698 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:28.393122 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:28.393040 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:28.393545 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:28.393171 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:28.393545 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:28.393040 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:28.393545 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:28.393529 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:29.393460 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:29.392978 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:29.393460 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:29.393128 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:29.932076 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:29.931301 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw64\" (UniqueName: \"kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64\") pod \"network-check-target-vr2l7\" (UID: \"afc23e34-857f-4c29-a381-cf87340044e7\") " pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:29.932076 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:29.931394 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs\") pod \"network-metrics-daemon-wg27k\" (UID: \"2e1b9d64-bc6b-4a12-86da-7cb47800704b\") " pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:29.932076 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:29.931564 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:29.932076 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:29.931634 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs podName:2e1b9d64-bc6b-4a12-86da-7cb47800704b nodeName:}" failed. No retries permitted until 2026-07-10 16:54:37.931616334 +0000 UTC m=+18.057324726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs") pod "network-metrics-daemon-wg27k" (UID: "2e1b9d64-bc6b-4a12-86da-7cb47800704b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:29.932076 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:29.931984 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:54:29.932076 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:29.932004 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:54:29.932076 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:29.932014 2559 projected.go:196] Error preparing data for projected volume kube-api-access-lqw64 for pod openshift-network-diagnostics/network-check-target-vr2l7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:29.932076 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:29.932053 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64 podName:afc23e34-857f-4c29-a381-cf87340044e7 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:37.932043235 +0000 UTC m=+18.057751623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lqw64" (UniqueName: "kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64") pod "network-check-target-vr2l7" (UID: "afc23e34-857f-4c29-a381-cf87340044e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:30.031800 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:30.031762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:30.031988 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:30.031952 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:30.032066 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:30.032030 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret podName:2a54dfc8-2f7b-4e80-9f23-03c128feeb24 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:38.032010428 +0000 UTC m=+18.157718829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret") pod "global-pull-secret-syncer-xdv2b" (UID: "2a54dfc8-2f7b-4e80-9f23-03c128feeb24") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:30.394592 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:30.393893 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:30.394592 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:30.394007 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:30.394592 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:30.394381 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:30.394592 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:30.394478 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:31.392904 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:31.392867 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:31.393076 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:31.392998 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:32.392896 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:32.392820 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:32.393359 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:32.392918 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:32.393359 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:32.393005 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:32.393359 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:32.393106 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:33.393286 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:33.392791 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:33.393286 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:33.392940 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:34.393389 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:34.393353 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:34.393879 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:34.393353 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:34.393879 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:34.393483 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:34.393879 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:34.393564 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:35.392833 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:35.392797 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:35.393014 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:35.392931 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:36.392801 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:36.392758 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:36.393282 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:36.392802 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:36.393282 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:36.392895 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:36.393282 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:36.392975 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:37.393147 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:37.393113 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:37.393666 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:37.393237 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:37.995788 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:37.995744 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw64\" (UniqueName: \"kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64\") pod \"network-check-target-vr2l7\" (UID: \"afc23e34-857f-4c29-a381-cf87340044e7\") " pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:37.995980 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:37.995828 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs\") pod \"network-metrics-daemon-wg27k\" (UID: \"2e1b9d64-bc6b-4a12-86da-7cb47800704b\") " pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:37.995980 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:37.995890 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:54:37.995980 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:37.995914 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:54:37.995980 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:37.995924 2559 projected.go:196] Error preparing data for projected volume kube-api-access-lqw64 for pod openshift-network-diagnostics/network-check-target-vr2l7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:37.995980 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:37.995970 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64 podName:afc23e34-857f-4c29-a381-cf87340044e7 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:53.995957549 +0000 UTC m=+34.121665936 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lqw64" (UniqueName: "kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64") pod "network-check-target-vr2l7" (UID: "afc23e34-857f-4c29-a381-cf87340044e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:37.995980 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:37.995971 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:37.996291 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:37.996026 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs podName:2e1b9d64-bc6b-4a12-86da-7cb47800704b nodeName:}" failed. No retries permitted until 2026-07-10 16:54:53.996013025 +0000 UTC m=+34.121721418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs") pod "network-metrics-daemon-wg27k" (UID: "2e1b9d64-bc6b-4a12-86da-7cb47800704b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:38.096846 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:38.096801 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:38.097037 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:38.096981 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:38.097103 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:38.097068 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret podName:2a54dfc8-2f7b-4e80-9f23-03c128feeb24 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:54.097046438 +0000 UTC m=+34.222754829 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret") pod "global-pull-secret-syncer-xdv2b" (UID: "2a54dfc8-2f7b-4e80-9f23-03c128feeb24") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:38.392656 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:38.392572 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:38.392808 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:38.392703 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:38.392899 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:38.392573 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:38.393034 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:38.392983 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:39.392765 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:39.392725 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:39.393178 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:39.392885 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:40.393403 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:40.393368 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:40.393833 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:40.393506 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:40.393833 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:40.393549 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:40.393833 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:40.393651 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:41.392726 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:41.392679 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:41.392915 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:41.392833 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:42.392588 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:42.392541 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:42.393138 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:42.392717 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:42.393138 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:42.393078 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:42.393255 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:42.393174 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:43.392667 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.392334 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:43.393307 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:43.392755 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:43.595310 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.595271 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4tdzm" event={"ID":"75c5040f-8551-4491-8725-e0b4e51a058a","Type":"ContainerStarted","Data":"e899e3205d794a15824c72654ed75cc93f51239a52a43e0e42703bb8c23fbcb0"} Jul 10 16:54:43.597421 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.597392 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q97mw" event={"ID":"e53d41a0-f680-4189-a2d1-0500d81ca3a9","Type":"ContainerStarted","Data":"8d351ef59fe6b95a27ac12a6291c5771308c2e9d34b40fae267c7f448110197b"} Jul 10 16:54:43.599326 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.599034 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bsmqb" event={"ID":"4346a8a8-2c9c-4d06-bb87-a5364485b29d","Type":"ContainerStarted","Data":"7796fc8cd0a6001036be2bdd49d011a39e4b0b33d3636608ed88d2c9c990a10a"} Jul 10 16:54:43.600861 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.600808 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vxt8w" event={"ID":"b7c2a0f7-7fe0-4084-9b87-6c1e95422355","Type":"ContainerStarted","Data":"c1b0b8590e852832ca31eee2969f32dc080e8774ad8de68ac031d32ad176a449"} Jul 10 16:54:43.602477 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.602452 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-grg9k" event={"ID":"a63b13f7-1bec-4127-a3ac-9aa89b83b424","Type":"ContainerStarted","Data":"ffe5a796f16df517eac8aac41830fcc078eb2a10fce63356cf118c201a54270c"} Jul 10 16:54:43.604173 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.604151 2559 generic.go:356] "Generic (PLEG): container finished" podID="1cf9f299-96ec-4148-bd09-9641e7d859f1" containerID="215fc17716650475736d0871f3a01e71d37f0dfd6032e891cf0cf766b0893863" exitCode=0 Jul 10 16:54:43.604279 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.604212 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxchh" event={"ID":"1cf9f299-96ec-4148-bd09-9641e7d859f1","Type":"ContainerDied","Data":"215fc17716650475736d0871f3a01e71d37f0dfd6032e891cf0cf766b0893863"} Jul 10 16:54:43.608041 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.608018 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" event={"ID":"dd754214-d46d-404b-8d29-699dcd61dd33","Type":"ContainerStarted","Data":"ea61902e176870b56d5cf6e351de91ed9d2f96e90adeb138ae66cd21dd52ea6e"} Jul 10 16:54:43.608131 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.608097 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" event={"ID":"dd754214-d46d-404b-8d29-699dcd61dd33","Type":"ContainerStarted","Data":"eed2c7e95b32ec731dc0928486ff1b1935342c2930670f9cd28ff11dd1d2b094"} Jul 10 16:54:43.608182 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.608149 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" event={"ID":"dd754214-d46d-404b-8d29-699dcd61dd33","Type":"ContainerStarted","Data":"61e82cdfed59ebf7d7248fb8129d81cc62f7b17d066de5108c3693e394a2f73a"} Jul 10 16:54:43.608227 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.608197 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" event={"ID":"dd754214-d46d-404b-8d29-699dcd61dd33","Type":"ContainerStarted","Data":"8bc1218b524a8afe67ac614ad6adee785d4e4dd782f2b9f42db2cfcd382adca5"} Jul 10 16:54:43.608276 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.608244 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" event={"ID":"dd754214-d46d-404b-8d29-699dcd61dd33","Type":"ContainerStarted","Data":"b15554ca3f1a7c350b1291b34329d61545a8eaa4f74552794d8f7d1b37c3caad"} Jul 10 16:54:43.608327 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.608274 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" event={"ID":"dd754214-d46d-404b-8d29-699dcd61dd33","Type":"ContainerStarted","Data":"09b50d14acd119894fb1e0bf89d706b4ec30bbef94022c4e7e700876f61f1272"} Jul 10 16:54:43.609646 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.609623 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" event={"ID":"5dc130ed-3574-4a74-aa2c-e20959046b96","Type":"ContainerStarted","Data":"a79d3ddabe401ba536b828d3e65cb83b91fa6a4d4013696e78d87d0182862cc6"} Jul 10 16:54:43.613108 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.613070 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4tdzm" podStartSLOduration=8.472118118 podStartE2EDuration="23.613058884s" totalImagesPullingTime="15.140940766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:43.612755279 +0000 UTC m=+23.738463690" watchObservedRunningTime="2026-07-10 16:54:43.613058884 +0000 UTC m=+23.738767295" Jul 10 16:54:43.613238 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.613198 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-228.ec2.internal" podStartSLOduration=21.613193761 podStartE2EDuration="21.613193761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:25.575402875 +0000 UTC m=+5.701111286" watchObservedRunningTime="2026-07-10 16:54:43.613193761 +0000 UTC m=+23.738902171" Jul 10 16:54:43.629563 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.629504 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bsmqb" podStartSLOduration=3.187803083 podStartE2EDuration="22.629470111s" totalImagesPullingTime="19.441667028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:43.629114186 +0000 UTC m=+23.754822597" watchObservedRunningTime="2026-07-10 16:54:43.629470111 +0000 UTC m=+23.755178520" Jul 10 16:54:43.665836 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.665756 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q97mw" podStartSLOduration=3.121429504 podStartE2EDuration="22.665735318s" totalImagesPullingTime="19.544305814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:43.664977433 +0000 UTC m=+23.790685843" watchObservedRunningTime="2026-07-10 16:54:43.665735318 +0000 UTC m=+23.791443729" Jul 10 16:54:43.685862 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.685778 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-grg9k" podStartSLOduration=3.213364392 podStartE2EDuration="22.685763524s" totalImagesPullingTime="19.472399132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:43.685397782 +0000 UTC m=+23.811106196" watchObservedRunningTime="2026-07-10 16:54:43.685763524 +0000 UTC m=+23.811471934" Jul 10 16:54:43.699173 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:43.699115 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vxt8w" podStartSLOduration=4.2312798 podStartE2EDuration="23.699097331s" totalImagesPullingTime="19.467817531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:43.699042105 +0000 UTC m=+23.824750514" watchObservedRunningTime="2026-07-10 16:54:43.699097331 +0000 UTC m=+23.824805741" Jul 10 16:54:44.393286 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:44.393256 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:44.393743 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:44.393368 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:44.393743 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:44.393427 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:44.393743 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:44.393551 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:44.446347 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:44.446316 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 16:54:44.614357 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:44.614266 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-78lb5" event={"ID":"cb371128-ff07-47e7-95a3-9815145b3724","Type":"ContainerStarted","Data":"651e25c45967e81a1f05208bf7192533326c62e763ed014888f91f1538fe35e2"} Jul 10 16:54:44.616429 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:44.616394 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" event={"ID":"5dc130ed-3574-4a74-aa2c-e20959046b96","Type":"ContainerStarted","Data":"c81e24a9ecf6834140db097005f5b650a27277835710fe1d41685044aba58dde"} Jul 10 16:54:44.628727 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:44.628666 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-78lb5" podStartSLOduration=5.162517659 podStartE2EDuration="24.628644856s" totalImagesPullingTime="19.466127197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:44.628412871 +0000 UTC m=+24.754121283" watchObservedRunningTime="2026-07-10 16:54:44.628644856 +0000 UTC m=+24.754353307" Jul 10 16:54:45.338822 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:45.338720 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T16:54:44.446337045Z","UUID":"6adfc3b4-400c-49e5-b5cb-4cc3f8ce5c02","Handler":null,"Name":"","Endpoint":""} Jul 10 16:54:45.341340 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:45.341311 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 16:54:45.341529 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:45.341379 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 16:54:45.392365 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:45.392326 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:45.392552 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:45.392447 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:46.393624 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:46.393357 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:46.394072 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:46.393417 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:46.394072 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:46.393732 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:46.394072 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:46.393828 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:46.624878 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:46.624824 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" event={"ID":"dd754214-d46d-404b-8d29-699dcd61dd33","Type":"ContainerStarted","Data":"213d5321ae36c8f6d23a11f1f2c6bd9c3f1028aadcdd398f63eab702056139b3"} Jul 10 16:54:46.627082 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:46.627059 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" event={"ID":"5dc130ed-3574-4a74-aa2c-e20959046b96","Type":"ContainerStarted","Data":"9293400b757ec249e5ae483ad87915905256bcb41ee9665f233a28ed1230632b"} Jul 10 16:54:46.645688 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:46.645588 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkpr7" podStartSLOduration=4.921398948 podStartE2EDuration="26.645572326s" totalImagesPullingTime="21.724173378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:46.644822196 +0000 UTC m=+26.770530609" watchObservedRunningTime="2026-07-10 16:54:46.645572326 +0000 UTC m=+26.771280777" Jul 10 16:54:46.953844 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:46.953773 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4tdzm" Jul 10 16:54:46.954509 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:46.954469 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4tdzm" Jul 10 16:54:47.137785 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:47.137749 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4tdzm" Jul 10 16:54:47.138289 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:47.138263 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4tdzm" Jul 10 16:54:47.392790 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:47.392715 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:47.392929 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:47.392836 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:48.393631 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:48.393398 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:48.394331 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:48.393465 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:48.394331 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:48.393668 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:48.394331 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:48.393742 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:48.634658 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:48.634612 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" event={"ID":"dd754214-d46d-404b-8d29-699dcd61dd33","Type":"ContainerStarted","Data":"f773df08425a0b2de22a17495f991d07341b53c57ba8dc8738341def6a2737b1"} Jul 10 16:54:48.634892 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:48.634867 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:48.635035 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:48.634933 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:48.636462 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:48.636439 2559 generic.go:356] "Generic (PLEG): container finished" podID="1cf9f299-96ec-4148-bd09-9641e7d859f1" containerID="e038c23200e6522fe57588d6e3606aadfa638520c05bc37f432083381a2ff03a" exitCode=0 Jul 10 16:54:48.636609 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:48.636521 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxchh" event={"ID":"1cf9f299-96ec-4148-bd09-9641e7d859f1","Type":"ContainerDied","Data":"e038c23200e6522fe57588d6e3606aadfa638520c05bc37f432083381a2ff03a"} Jul 10 16:54:48.651080 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:48.651009 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:48.660752 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:48.660708 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" podStartSLOduration=8.154577973 podStartE2EDuration="27.66069459s" totalImagesPullingTime="19.506116617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:48.659199721 +0000 UTC m=+28.784908130" watchObservedRunningTime="2026-07-10 16:54:48.66069459 +0000 UTC m=+28.786403044" Jul 10 16:54:49.393043 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:49.392558 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:49.393043 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:49.392700 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:49.639605 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:49.639567 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:49.655329 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:49.655264 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:54:49.960120 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:49.960094 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xdv2b"] Jul 10 16:54:49.960263 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:49.960233 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:49.960343 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:49.960322 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:49.963794 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:49.963761 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vr2l7"] Jul 10 16:54:49.963926 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:49.963912 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:49.964072 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:49.964036 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:49.964451 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:49.964427 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wg27k"] Jul 10 16:54:49.964570 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:49.964552 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:49.964699 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:49.964679 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:50.643204 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:50.643023 2559 generic.go:356] "Generic (PLEG): container finished" podID="1cf9f299-96ec-4148-bd09-9641e7d859f1" containerID="6679f4bfbd891e981de9f59da50b6f5a41e896b4ebd579976d385a0cbacf6b7f" exitCode=0 Jul 10 16:54:50.643693 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:50.643101 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxchh" event={"ID":"1cf9f299-96ec-4148-bd09-9641e7d859f1","Type":"ContainerDied","Data":"6679f4bfbd891e981de9f59da50b6f5a41e896b4ebd579976d385a0cbacf6b7f"} Jul 10 16:54:51.393144 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:51.393101 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:51.393331 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:51.393101 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:51.393331 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:51.393211 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:51.393331 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:51.393306 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:52.393342 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:52.393266 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:52.393828 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:52.393415 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:52.550727 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:52.550695 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:52.653633 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:52.653598 2559 generic.go:356] "Generic (PLEG): container finished" podID="1cf9f299-96ec-4148-bd09-9641e7d859f1" containerID="a7acaf55ac0e6e323d85ea9dc045089979e6e094502d92308d9fc0cc98356951" exitCode=0 Jul 10 16:54:52.653790 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:52.653680 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxchh" event={"ID":"1cf9f299-96ec-4148-bd09-9641e7d859f1","Type":"ContainerDied","Data":"a7acaf55ac0e6e323d85ea9dc045089979e6e094502d92308d9fc0cc98356951"} Jul 10 16:54:52.664427 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:52.664398 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:53.392580 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:53.392544 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:53.392762 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:53.392559 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:53.392762 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:53.392669 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:53.392762 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:53.392717 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:54.022598 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:54.022551 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs\") pod \"network-metrics-daemon-wg27k\" (UID: \"2e1b9d64-bc6b-4a12-86da-7cb47800704b\") " pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:54.023206 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:54.022641 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw64\" (UniqueName: \"kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64\") pod \"network-check-target-vr2l7\" (UID: \"afc23e34-857f-4c29-a381-cf87340044e7\") " pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:54.023206 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:54.022710 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:54.023206 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:54.022776 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:54:54.023206 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:54.022786 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs podName:2e1b9d64-bc6b-4a12-86da-7cb47800704b nodeName:}" failed. No retries permitted until 2026-07-10 16:55:26.022765372 +0000 UTC m=+66.148473761 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs") pod "network-metrics-daemon-wg27k" (UID: "2e1b9d64-bc6b-4a12-86da-7cb47800704b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:54:54.023206 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:54.022795 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:54:54.023206 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:54.022805 2559 projected.go:196] Error preparing data for projected volume kube-api-access-lqw64 for pod openshift-network-diagnostics/network-check-target-vr2l7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:54.023206 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:54.022841 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64 podName:afc23e34-857f-4c29-a381-cf87340044e7 nodeName:}" failed. No retries permitted until 2026-07-10 16:55:26.022830723 +0000 UTC m=+66.148539110 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lqw64" (UniqueName: "kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64") pod "network-check-target-vr2l7" (UID: "afc23e34-857f-4c29-a381-cf87340044e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:54:54.028007 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:54.027979 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:54.123575 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:54.123532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:54.123752 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:54.123719 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:54.123813 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:54.123800 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret podName:2a54dfc8-2f7b-4e80-9f23-03c128feeb24 nodeName:}" failed. No retries permitted until 2026-07-10 16:55:26.123780869 +0000 UTC m=+66.249489262 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret") pod "global-pull-secret-syncer-xdv2b" (UID: "2a54dfc8-2f7b-4e80-9f23-03c128feeb24") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:54:54.126254 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:54.126222 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:54.392870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:54.392791 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:54.393038 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:54.392936 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:55.393184 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:55.393152 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:55.393572 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:55.393163 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:55.393572 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:55.393267 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdv2b" podUID="2a54dfc8-2f7b-4e80-9f23-03c128feeb24" Jul 10 16:54:55.393572 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:55.393338 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vr2l7" podUID="afc23e34-857f-4c29-a381-cf87340044e7" Jul 10 16:54:55.919285 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:55.919239 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:56.020059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.020019 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:56.115954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.115927 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:56.216866 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.216780 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:56.317511 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.317463 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:56.392735 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.392696 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:56.392906 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:56.392840 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wg27k" podUID="2e1b9d64-bc6b-4a12-86da-7cb47800704b" Jul 10 16:54:56.722195 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.722113 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-228.ec2.internal" event="NodeReady" Jul 10 16:54:56.722615 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.722289 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 16:54:56.763862 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.763828 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cd76bfb47-ktthd"] Jul 10 16:54:56.767435 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.767402 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.770079 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.770051 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 16:54:56.770202 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.770174 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-phvw6\"" Jul 10 16:54:56.770519 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.770483 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 16:54:56.770922 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.770757 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 16:54:56.778956 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.778935 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lmdzg"] Jul 10 16:54:56.781288 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.781268 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:56.783193 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.783172 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 16:54:56.784524 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.784487 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 16:54:56.784637 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.784543 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 16:54:56.784637 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.784572 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ccqfl\"" Jul 10 16:54:56.787775 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.787715 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cd76bfb47-ktthd"] Jul 10 16:54:56.788433 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.788407 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-62nvr"] Jul 10 16:54:56.791105 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.791086 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62nvr" Jul 10 16:54:56.793823 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.793803 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lmdzg"] Jul 10 16:54:56.795988 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.795966 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 16:54:56.796232 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.796207 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b5v82\"" Jul 10 16:54:56.796317 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.796255 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 16:54:56.796534 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.796513 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 16:54:56.802121 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.802100 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62nvr"] Jul 10 16:54:56.843897 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.843859 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-certificates\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.844083 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.843908 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.844083 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.843934 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4edf2bfe-ec0e-4469-8401-8f42470ade3b-trusted-ca\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.844083 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.844023 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4edf2bfe-ec0e-4469-8401-8f42470ade3b-image-registry-private-configuration\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.844083 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.844064 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4edf2bfe-ec0e-4469-8401-8f42470ade3b-installation-pull-secrets\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.844252 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.844118 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-bound-sa-token\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.844252 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.844142 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4edf2bfe-ec0e-4469-8401-8f42470ade3b-ca-trust-extracted\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.844252 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.844165 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk6lv\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-kube-api-access-vk6lv\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.944696 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.944662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-bound-sa-token\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.944696 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.944704 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn6sz\" (UniqueName: \"kubernetes.io/projected/31fb5d73-6dcf-4524-b6d8-cefa582aa432-kube-api-access-wn6sz\") pod \"ingress-canary-62nvr\" (UID: \"31fb5d73-6dcf-4524-b6d8-cefa582aa432\") " pod="openshift-ingress-canary/ingress-canary-62nvr" Jul 10 16:54:56.944893 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.944724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4edf2bfe-ec0e-4469-8401-8f42470ade3b-ca-trust-extracted\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.944893 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.944771 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk6lv\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-kube-api-access-vk6lv\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.944990 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.944967 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-certificates\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.945039 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.945006 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d79cf9-d7b1-4e10-ab4e-59c351659657-tmp-dir\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:56.945112 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.945090 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4edf2bfe-ec0e-4469-8401-8f42470ade3b-ca-trust-extracted\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.945156 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.945094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.945191 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.945159 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4edf2bfe-ec0e-4469-8401-8f42470ade3b-trusted-ca\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.945304 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.945191 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:56.945304 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:56.945163 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 16:54:56.945304 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.945233 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4edf2bfe-ec0e-4469-8401-8f42470ade3b-image-registry-private-configuration\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.945304 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:56.945250 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd76bfb47-ktthd: secret "image-registry-tls" not found Jul 10 16:54:56.945611 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:56.945316 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls podName:4edf2bfe-ec0e-4469-8401-8f42470ade3b nodeName:}" failed. No retries permitted until 2026-07-10 16:54:57.445292912 +0000 UTC m=+37.571001316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls") pod "image-registry-7cd76bfb47-ktthd" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b") : secret "image-registry-tls" not found Jul 10 16:54:56.945611 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.945347 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4edf2bfe-ec0e-4469-8401-8f42470ade3b-installation-pull-secrets\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.945611 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.945417 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31fb5d73-6dcf-4524-b6d8-cefa582aa432-cert\") pod \"ingress-canary-62nvr\" (UID: \"31fb5d73-6dcf-4524-b6d8-cefa582aa432\") " pod="openshift-ingress-canary/ingress-canary-62nvr" Jul 10 16:54:56.945611 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.945443 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d79cf9-d7b1-4e10-ab4e-59c351659657-config-volume\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:56.945611 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.945471 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcllr\" (UniqueName: \"kubernetes.io/projected/f0d79cf9-d7b1-4e10-ab4e-59c351659657-kube-api-access-gcllr\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:56.945611 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.945595 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-certificates\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.946116 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.946091 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4edf2bfe-ec0e-4469-8401-8f42470ade3b-trusted-ca\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.949801 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.949776 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4edf2bfe-ec0e-4469-8401-8f42470ade3b-installation-pull-secrets\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.949801 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.949796 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4edf2bfe-ec0e-4469-8401-8f42470ade3b-image-registry-private-configuration\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.954433 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.954409 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk6lv\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-kube-api-access-vk6lv\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.954684 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.954665 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-bound-sa-token\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:56.959999 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:56.959959 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:57.045936 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.045915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31fb5d73-6dcf-4524-b6d8-cefa582aa432-cert\") pod \"ingress-canary-62nvr\" (UID: \"31fb5d73-6dcf-4524-b6d8-cefa582aa432\") " pod="openshift-ingress-canary/ingress-canary-62nvr" Jul 10 16:54:57.045987 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.045947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d79cf9-d7b1-4e10-ab4e-59c351659657-config-volume\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:57.045987 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.045973 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcllr\" (UniqueName: \"kubernetes.io/projected/f0d79cf9-d7b1-4e10-ab4e-59c351659657-kube-api-access-gcllr\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:57.046073 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.045995 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6sz\" (UniqueName: \"kubernetes.io/projected/31fb5d73-6dcf-4524-b6d8-cefa582aa432-kube-api-access-wn6sz\") pod \"ingress-canary-62nvr\" (UID: \"31fb5d73-6dcf-4524-b6d8-cefa582aa432\") " pod="openshift-ingress-canary/ingress-canary-62nvr" Jul 10 16:54:57.046073 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.046040 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d79cf9-d7b1-4e10-ab4e-59c351659657-tmp-dir\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:57.046167 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.046078 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:57.046223 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:57.046175 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:54:57.046267 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:57.046234 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls podName:f0d79cf9-d7b1-4e10-ab4e-59c351659657 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:57.546215755 +0000 UTC m=+37.671924145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls") pod "dns-default-lmdzg" (UID: "f0d79cf9-d7b1-4e10-ab4e-59c351659657") : secret "dns-default-metrics-tls" not found Jul 10 16:54:57.046417 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.046393 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d79cf9-d7b1-4e10-ab4e-59c351659657-tmp-dir\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:57.046664 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.046645 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d79cf9-d7b1-4e10-ab4e-59c351659657-config-volume\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:57.048976 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.048948 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31fb5d73-6dcf-4524-b6d8-cefa582aa432-cert\") pod \"ingress-canary-62nvr\" (UID: \"31fb5d73-6dcf-4524-b6d8-cefa582aa432\") " pod="openshift-ingress-canary/ingress-canary-62nvr" Jul 10 16:54:57.058188 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.058165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6sz\" (UniqueName: \"kubernetes.io/projected/31fb5d73-6dcf-4524-b6d8-cefa582aa432-kube-api-access-wn6sz\") pod \"ingress-canary-62nvr\" (UID: \"31fb5d73-6dcf-4524-b6d8-cefa582aa432\") " pod="openshift-ingress-canary/ingress-canary-62nvr" Jul 10 16:54:57.058308 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.058225 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcllr\" (UniqueName: \"kubernetes.io/projected/f0d79cf9-d7b1-4e10-ab4e-59c351659657-kube-api-access-gcllr\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:57.058685 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.058665 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:57.102093 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.102049 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62nvr" Jul 10 16:54:57.240206 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.240173 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62nvr"] Jul 10 16:54:57.240963 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:54:57.240936 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31fb5d73_6dcf_4524_b6d8_cefa582aa432.slice/crio-72bfef15a6ec61cb644da9d328928db4761ec6dd69d5bdcc6071d96e08ea018d WatchSource:0}: Error finding container 72bfef15a6ec61cb644da9d328928db4761ec6dd69d5bdcc6071d96e08ea018d: Status 404 returned error can't find the container with id 72bfef15a6ec61cb644da9d328928db4761ec6dd69d5bdcc6071d96e08ea018d Jul 10 16:54:57.392691 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.392654 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:54:57.392849 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.392823 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:54:57.400035 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.400015 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 16:54:57.400237 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.400184 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 16:54:57.400363 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.400315 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 16:54:57.400825 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.400810 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jn8d4\"" Jul 10 16:54:57.448076 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.448038 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:57.448227 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:57.448190 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 16:54:57.448227 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:57.448206 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd76bfb47-ktthd: secret "image-registry-tls" not found Jul 10 16:54:57.448296 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:57.448267 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls podName:4edf2bfe-ec0e-4469-8401-8f42470ade3b nodeName:}" failed. No retries permitted until 2026-07-10 16:54:58.448252074 +0000 UTC m=+38.573960461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls") pod "image-registry-7cd76bfb47-ktthd" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b") : secret "image-registry-tls" not found Jul 10 16:54:57.549391 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.549299 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:57.549559 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:57.549423 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:54:57.549559 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:57.549478 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls podName:f0d79cf9-d7b1-4e10-ab4e-59c351659657 nodeName:}" failed. No retries permitted until 2026-07-10 16:54:58.549463781 +0000 UTC m=+38.675172170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls") pod "dns-default-lmdzg" (UID: "f0d79cf9-d7b1-4e10-ab4e-59c351659657") : secret "dns-default-metrics-tls" not found Jul 10 16:54:57.668925 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.668894 2559 generic.go:356] "Generic (PLEG): container finished" podID="1cf9f299-96ec-4148-bd09-9641e7d859f1" containerID="e0571f3412e9308f4d77afac507f544b0ae58021f5329de6e914279440a7b3dd" exitCode=0 Jul 10 16:54:57.669107 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.668979 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxchh" event={"ID":"1cf9f299-96ec-4148-bd09-9641e7d859f1","Type":"ContainerDied","Data":"e0571f3412e9308f4d77afac507f544b0ae58021f5329de6e914279440a7b3dd"} Jul 10 16:54:57.670448 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.670052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62nvr" event={"ID":"31fb5d73-6dcf-4524-b6d8-cefa582aa432","Type":"ContainerStarted","Data":"72bfef15a6ec61cb644da9d328928db4761ec6dd69d5bdcc6071d96e08ea018d"} Jul 10 16:54:57.680319 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.680294 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:57.744780 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:57.744747 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:58.393365 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:58.393070 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:54:58.396180 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:58.396153 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdmq6\"" Jul 10 16:54:58.396290 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:58.396238 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 16:54:58.458435 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:58.458372 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:54:58.458644 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:58.458564 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 16:54:58.458644 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:58.458588 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd76bfb47-ktthd: secret "image-registry-tls" not found Jul 10 16:54:58.458749 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:58.458653 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls podName:4edf2bfe-ec0e-4469-8401-8f42470ade3b nodeName:}" failed. No retries permitted until 2026-07-10 16:55:00.458633201 +0000 UTC m=+40.584341589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls") pod "image-registry-7cd76bfb47-ktthd" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b") : secret "image-registry-tls" not found Jul 10 16:54:58.559563 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:58.559525 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:54:58.559729 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:58.559671 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:54:58.559766 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:54:58.559734 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls podName:f0d79cf9-d7b1-4e10-ab4e-59c351659657 nodeName:}" failed. No retries permitted until 2026-07-10 16:55:00.559718685 +0000 UTC m=+40.685427073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls") pod "dns-default-lmdzg" (UID: "f0d79cf9-d7b1-4e10-ab4e-59c351659657") : secret "dns-default-metrics-tls" not found Jul 10 16:54:58.675885 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:58.675813 2559 generic.go:356] "Generic (PLEG): container finished" podID="1cf9f299-96ec-4148-bd09-9641e7d859f1" containerID="8d64dc05d153338a2a9a445caf680dc788aa626a97470462b2fbbb446dd67064" exitCode=0 Jul 10 16:54:58.675885 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:58.675861 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxchh" event={"ID":"1cf9f299-96ec-4148-bd09-9641e7d859f1","Type":"ContainerDied","Data":"8d64dc05d153338a2a9a445caf680dc788aa626a97470462b2fbbb446dd67064"} Jul 10 16:54:59.056917 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:59.056832 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:59.683845 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:59.683754 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxchh" event={"ID":"1cf9f299-96ec-4148-bd09-9641e7d859f1","Type":"ContainerStarted","Data":"c48af2e62d967dbca78f20803bcb3127682f84f01ee39e342be2e54d2ac2364c"} Jul 10 16:54:59.685199 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:59.685174 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62nvr" event={"ID":"31fb5d73-6dcf-4524-b6d8-cefa582aa432","Type":"ContainerStarted","Data":"217b89877a486838158630583a672c6507ce590a8f69f4824242831cc75d5b7e"} Jul 10 16:54:59.714354 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:59.714300 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xxchh" podStartSLOduration=9.382263779 podStartE2EDuration="39.714282989s" totalImagesPullingTime="30.33201921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:59.711449102 +0000 UTC m=+39.837157548" watchObservedRunningTime="2026-07-10 16:54:59.714282989 +0000 UTC m=+39.839991400" Jul 10 16:54:59.726435 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:59.726378 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-62nvr" podStartSLOduration=1.466931947 podStartE2EDuration="3.726359209s" totalImagesPullingTime="2.259427262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:54:59.726321072 +0000 UTC m=+39.852029482" watchObservedRunningTime="2026-07-10 16:54:59.726359209 +0000 UTC m=+39.852067616" Jul 10 16:54:59.908258 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:59.908226 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:54:59.975157 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:54:59.975066 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:00.474270 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:00.474234 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:55:00.474915 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:00.474358 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 16:55:00.474915 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:00.474370 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd76bfb47-ktthd: secret "image-registry-tls" not found Jul 10 16:55:00.474915 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:00.474430 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls podName:4edf2bfe-ec0e-4469-8401-8f42470ade3b nodeName:}" failed. No retries permitted until 2026-07-10 16:55:04.474416477 +0000 UTC m=+44.600124865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls") pod "image-registry-7cd76bfb47-ktthd" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b") : secret "image-registry-tls" not found Jul 10 16:55:00.575575 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:00.575473 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:55:00.575757 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:00.575671 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:55:00.575795 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:00.575771 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls podName:f0d79cf9-d7b1-4e10-ab4e-59c351659657 nodeName:}" failed. No retries permitted until 2026-07-10 16:55:04.575748898 +0000 UTC m=+44.701457309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls") pod "dns-default-lmdzg" (UID: "f0d79cf9-d7b1-4e10-ab4e-59c351659657") : secret "dns-default-metrics-tls" not found Jul 10 16:55:00.748016 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:00.747986 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:00.826810 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:00.826706 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:01.962081 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:01.962045 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:02.018104 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:02.018060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:03.431777 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.431746 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:03.490206 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.490160 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:03.688699 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.688538 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b4frc"] Jul 10 16:55:03.697318 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.697291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:03.719420 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.719381 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b4frc"] Jul 10 16:55:03.719565 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.719528 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:03.722116 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.722092 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 16:55:03.722271 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.722150 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 16:55:03.722336 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.722156 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 16:55:03.722336 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.722158 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vsnsq\"" Jul 10 16:55:03.722444 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.722200 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 16:55:03.850811 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.850774 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:03.898165 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.898135 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de9e01f8-0325-4387-a668-8dc75d3e13bc-data-volume\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:03.898165 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.898167 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:03.898342 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.898312 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnt5w\" (UniqueName: \"kubernetes.io/projected/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-api-access-dnt5w\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:03.898405 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.898385 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:03.898436 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.898418 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de9e01f8-0325-4387-a668-8dc75d3e13bc-crio-socket\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:03.934630 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.934596 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:03.999433 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.999354 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:03.999433 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.999397 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de9e01f8-0325-4387-a668-8dc75d3e13bc-crio-socket\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:03.999433 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.999421 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de9e01f8-0325-4387-a668-8dc75d3e13bc-data-volume\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:03.999433 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.999436 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:03.999734 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:03.999539 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:55:03.999734 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.999571 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnt5w\" (UniqueName: \"kubernetes.io/projected/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-api-access-dnt5w\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:03.999734 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:03.999592 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:55:03.999734 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:03.999630 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls podName:de9e01f8-0325-4387-a668-8dc75d3e13bc nodeName:}" failed. No retries permitted until 2026-07-10 16:55:04.499607418 +0000 UTC m=+44.625315806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4frc" (UID: "de9e01f8-0325-4387-a668-8dc75d3e13bc") : secret "insights-runtime-extractor-tls" not found Jul 10 16:55:03.999734 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.999644 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de9e01f8-0325-4387-a668-8dc75d3e13bc-crio-socket\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:03.999734 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:03.999668 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm podName:de9e01f8-0325-4387-a668-8dc75d3e13bc nodeName:}" failed. No retries permitted until 2026-07-10 16:55:04.499651956 +0000 UTC m=+44.625360367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b4frc" (UID: "de9e01f8-0325-4387-a668-8dc75d3e13bc") : configmap "kube-rbac-proxy" not found Jul 10 16:55:03.999946 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:03.999802 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de9e01f8-0325-4387-a668-8dc75d3e13bc-data-volume\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:04.009534 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.009480 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnt5w\" (UniqueName: \"kubernetes.io/projected/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-api-access-dnt5w\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:04.143409 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.143378 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:04.244889 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.244853 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:04.502573 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.502539 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:04.502943 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.502610 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:55:04.502943 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.502654 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:04.502943 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:04.502679 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:55:04.502943 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:04.502695 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 16:55:04.502943 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:04.502708 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd76bfb47-ktthd: secret "image-registry-tls" not found Jul 10 16:55:04.502943 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:04.502742 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:55:04.502943 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:04.502759 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls podName:4edf2bfe-ec0e-4469-8401-8f42470ade3b nodeName:}" failed. No retries permitted until 2026-07-10 16:55:12.502743951 +0000 UTC m=+52.628452341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls") pod "image-registry-7cd76bfb47-ktthd" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b") : secret "image-registry-tls" not found Jul 10 16:55:04.502943 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:04.502777 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls podName:de9e01f8-0325-4387-a668-8dc75d3e13bc nodeName:}" failed. No retries permitted until 2026-07-10 16:55:05.50276613 +0000 UTC m=+45.628474518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4frc" (UID: "de9e01f8-0325-4387-a668-8dc75d3e13bc") : secret "insights-runtime-extractor-tls" not found Jul 10 16:55:04.502943 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:04.502789 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm podName:de9e01f8-0325-4387-a668-8dc75d3e13bc nodeName:}" failed. No retries permitted until 2026-07-10 16:55:05.502782758 +0000 UTC m=+45.628491146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b4frc" (UID: "de9e01f8-0325-4387-a668-8dc75d3e13bc") : configmap "kube-rbac-proxy" not found Jul 10 16:55:04.603004 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.602962 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:55:04.603149 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:04.603082 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:55:04.603149 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:04.603139 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls podName:f0d79cf9-d7b1-4e10-ab4e-59c351659657 nodeName:}" failed. No retries permitted until 2026-07-10 16:55:12.603123093 +0000 UTC m=+52.728831482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls") pod "dns-default-lmdzg" (UID: "f0d79cf9-d7b1-4e10-ab4e-59c351659657") : secret "dns-default-metrics-tls" not found Jul 10 16:55:04.804671 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.804585 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bsxw"] Jul 10 16:55:04.828401 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.828371 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bsxw"] Jul 10 16:55:04.828577 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.828483 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bsxw" Jul 10 16:55:04.830926 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.830908 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 16:55:04.831019 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.830964 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 16:55:04.831997 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.831976 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qxmk7\"" Jul 10 16:55:04.837847 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.837827 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:04.900632 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:04.900597 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:05.006515 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.006459 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6npzk\" (UniqueName: \"kubernetes.io/projected/c9c041b9-a688-4931-9b40-7d9baf3a3254-kube-api-access-6npzk\") pod \"migrator-7f59b54f6c-7bsxw\" (UID: \"c9c041b9-a688-4931-9b40-7d9baf3a3254\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bsxw" Jul 10 16:55:05.107860 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.107826 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6npzk\" (UniqueName: \"kubernetes.io/projected/c9c041b9-a688-4931-9b40-7d9baf3a3254-kube-api-access-6npzk\") pod \"migrator-7f59b54f6c-7bsxw\" (UID: \"c9c041b9-a688-4931-9b40-7d9baf3a3254\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bsxw" Jul 10 16:55:05.116959 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.116929 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6npzk\" (UniqueName: \"kubernetes.io/projected/c9c041b9-a688-4931-9b40-7d9baf3a3254-kube-api-access-6npzk\") pod \"migrator-7f59b54f6c-7bsxw\" (UID: \"c9c041b9-a688-4931-9b40-7d9baf3a3254\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bsxw" Jul 10 16:55:05.136934 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.136898 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bsxw" Jul 10 16:55:05.284160 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.284043 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bsxw"] Jul 10 16:55:05.286809 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:05.286782 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c041b9_a688_4931_9b40_7d9baf3a3254.slice/crio-428d48e8eb8a5e1ca75d96eadfa5399ac59e1716f9ab10a9738f9d6458c71166 WatchSource:0}: Error finding container 428d48e8eb8a5e1ca75d96eadfa5399ac59e1716f9ab10a9738f9d6458c71166: Status 404 returned error can't find the container with id 428d48e8eb8a5e1ca75d96eadfa5399ac59e1716f9ab10a9738f9d6458c71166 Jul 10 16:55:05.474368 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.474290 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vxt8w_b7c2a0f7-7fe0-4084-9b87-6c1e95422355/dns-node-resolver/0.log" Jul 10 16:55:05.512171 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.512134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:05.512551 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.512231 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:05.512551 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:05.512285 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:55:05.512551 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:05.512340 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:55:05.512551 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:05.512347 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls podName:de9e01f8-0325-4387-a668-8dc75d3e13bc nodeName:}" failed. No retries permitted until 2026-07-10 16:55:07.512331412 +0000 UTC m=+47.638039800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4frc" (UID: "de9e01f8-0325-4387-a668-8dc75d3e13bc") : secret "insights-runtime-extractor-tls" not found Jul 10 16:55:05.512551 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:05.512401 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm podName:de9e01f8-0325-4387-a668-8dc75d3e13bc nodeName:}" failed. No retries permitted until 2026-07-10 16:55:07.512382992 +0000 UTC m=+47.638091380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b4frc" (UID: "de9e01f8-0325-4387-a668-8dc75d3e13bc") : configmap "kube-rbac-proxy" not found Jul 10 16:55:05.701045 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.701010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bsxw" event={"ID":"c9c041b9-a688-4931-9b40-7d9baf3a3254","Type":"ContainerStarted","Data":"428d48e8eb8a5e1ca75d96eadfa5399ac59e1716f9ab10a9738f9d6458c71166"} Jul 10 16:55:05.736859 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.736776 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-dvsbj"] Jul 10 16:55:05.764066 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.764024 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-dvsbj"] Jul 10 16:55:05.764225 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.764149 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" Jul 10 16:55:05.766838 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.766810 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 16:55:05.766965 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.766875 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 16:55:05.767864 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.767841 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 16:55:05.768016 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.767888 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2k8tb\"" Jul 10 16:55:05.768016 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.767906 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 16:55:05.844878 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.844846 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:05.915395 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.915359 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/50d805bd-c66d-4919-9dc2-a25330819857-signing-cabundle\") pod \"service-ca-79f68b9f4f-dvsbj\" (UID: \"50d805bd-c66d-4919-9dc2-a25330819857\") " pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" Jul 10 16:55:05.915618 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.915558 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67n4h\" (UniqueName: \"kubernetes.io/projected/50d805bd-c66d-4919-9dc2-a25330819857-kube-api-access-67n4h\") pod \"service-ca-79f68b9f4f-dvsbj\" (UID: \"50d805bd-c66d-4919-9dc2-a25330819857\") " pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" Jul 10 16:55:05.915618 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.915608 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/50d805bd-c66d-4919-9dc2-a25330819857-signing-key\") pod \"service-ca-79f68b9f4f-dvsbj\" (UID: \"50d805bd-c66d-4919-9dc2-a25330819857\") " pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" Jul 10 16:55:05.945133 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:05.945105 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:06.016302 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:06.016214 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67n4h\" (UniqueName: \"kubernetes.io/projected/50d805bd-c66d-4919-9dc2-a25330819857-kube-api-access-67n4h\") pod \"service-ca-79f68b9f4f-dvsbj\" (UID: \"50d805bd-c66d-4919-9dc2-a25330819857\") " pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" Jul 10 16:55:06.016302 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:06.016254 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/50d805bd-c66d-4919-9dc2-a25330819857-signing-key\") pod \"service-ca-79f68b9f4f-dvsbj\" (UID: \"50d805bd-c66d-4919-9dc2-a25330819857\") " pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" Jul 10 16:55:06.016486 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:06.016304 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/50d805bd-c66d-4919-9dc2-a25330819857-signing-cabundle\") pod \"service-ca-79f68b9f4f-dvsbj\" (UID: \"50d805bd-c66d-4919-9dc2-a25330819857\") " pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" Jul 10 16:55:06.017031 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:06.017015 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/50d805bd-c66d-4919-9dc2-a25330819857-signing-cabundle\") pod \"service-ca-79f68b9f4f-dvsbj\" (UID: \"50d805bd-c66d-4919-9dc2-a25330819857\") " pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" Jul 10 16:55:06.018935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:06.018911 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/50d805bd-c66d-4919-9dc2-a25330819857-signing-key\") pod \"service-ca-79f68b9f4f-dvsbj\" (UID: \"50d805bd-c66d-4919-9dc2-a25330819857\") " pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" Jul 10 16:55:06.026638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:06.026613 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67n4h\" (UniqueName: \"kubernetes.io/projected/50d805bd-c66d-4919-9dc2-a25330819857-kube-api-access-67n4h\") pod \"service-ca-79f68b9f4f-dvsbj\" (UID: \"50d805bd-c66d-4919-9dc2-a25330819857\") " pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" Jul 10 16:55:06.073567 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:06.073531 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" Jul 10 16:55:06.074919 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:06.074898 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bsmqb_4346a8a8-2c9c-4d06-bb87-a5364485b29d/node-ca/0.log" Jul 10 16:55:06.249543 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:06.249508 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-dvsbj"] Jul 10 16:55:06.252905 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:06.252874 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d805bd_c66d_4919_9dc2_a25330819857.slice/crio-272ea98a0df48d99186cb22fdb7406383784a88dbe64b7cd690882bf8ebf5018 WatchSource:0}: Error finding container 272ea98a0df48d99186cb22fdb7406383784a88dbe64b7cd690882bf8ebf5018: Status 404 returned error can't find the container with id 272ea98a0df48d99186cb22fdb7406383784a88dbe64b7cd690882bf8ebf5018 Jul 10 16:55:06.704923 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:06.704884 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" event={"ID":"50d805bd-c66d-4919-9dc2-a25330819857","Type":"ContainerStarted","Data":"272ea98a0df48d99186cb22fdb7406383784a88dbe64b7cd690882bf8ebf5018"} Jul 10 16:55:06.879214 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:06.879184 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-62nvr_31fb5d73-6dcf-4524-b6d8-cefa582aa432/serve-healthcheck-canary/0.log" Jul 10 16:55:06.950133 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:06.950098 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:07.529446 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:07.529352 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:07.529446 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:07.529426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:07.529703 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:07.529503 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:55:07.529703 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:07.529534 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:55:07.529703 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:07.529584 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls podName:de9e01f8-0325-4387-a668-8dc75d3e13bc nodeName:}" failed. No retries permitted until 2026-07-10 16:55:11.52956369 +0000 UTC m=+51.655272083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4frc" (UID: "de9e01f8-0325-4387-a668-8dc75d3e13bc") : secret "insights-runtime-extractor-tls" not found Jul 10 16:55:07.529703 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:07.529603 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm podName:de9e01f8-0325-4387-a668-8dc75d3e13bc nodeName:}" failed. No retries permitted until 2026-07-10 16:55:11.529595508 +0000 UTC m=+51.655303897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b4frc" (UID: "de9e01f8-0325-4387-a668-8dc75d3e13bc") : configmap "kube-rbac-proxy" not found Jul 10 16:55:07.709617 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:07.709571 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bsxw" event={"ID":"c9c041b9-a688-4931-9b40-7d9baf3a3254","Type":"ContainerStarted","Data":"ef6445b792dbf1a7f330b7e9c4522f23e29e35d8e9babbc079159adaed75191c"} Jul 10 16:55:07.710063 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:07.709641 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bsxw" event={"ID":"c9c041b9-a688-4931-9b40-7d9baf3a3254","Type":"ContainerStarted","Data":"bd0c17be2d0ac0bfcf1336d63cb4d4031c89b40e03b643226e7a3fc804d9c2ca"} Jul 10 16:55:07.727842 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:07.727670 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bsxw" podStartSLOduration=1.787397528 podStartE2EDuration="3.727647797s" totalImagesPullingTime="1.940250269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:07.727032838 +0000 UTC m=+47.852741249" watchObservedRunningTime="2026-07-10 16:55:07.727647797 +0000 UTC m=+47.853356208" Jul 10 16:55:08.347442 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:08.347394 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:09.717299 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:09.717211 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" event={"ID":"50d805bd-c66d-4919-9dc2-a25330819857","Type":"ContainerStarted","Data":"c769cd4a8b75ad403cdb765ef8fa1751529ee66932c53c676e22380d884cb011"} Jul 10 16:55:09.733612 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:09.733565 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-dvsbj" podStartSLOduration=1.624032272 podStartE2EDuration="4.733546657s" totalImagesPullingTime="3.109514385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:09.733004177 +0000 UTC m=+49.858712588" watchObservedRunningTime="2026-07-10 16:55:09.733546657 +0000 UTC m=+49.859255066" Jul 10 16:55:10.158878 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:10.158847 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:11.037284 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:11.037234 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:11.357087 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:11.357052 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:11.460673 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:11.460634 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:11.562720 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:11.562680 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:11.562926 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:11.562764 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:11.562926 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:11.562809 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:55:11.562926 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:11.562881 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls podName:de9e01f8-0325-4387-a668-8dc75d3e13bc nodeName:}" failed. No retries permitted until 2026-07-10 16:55:19.562860542 +0000 UTC m=+59.688568943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4frc" (UID: "de9e01f8-0325-4387-a668-8dc75d3e13bc") : secret "insights-runtime-extractor-tls" not found Jul 10 16:55:11.563394 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:11.563372 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de9e01f8-0325-4387-a668-8dc75d3e13bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:11.605210 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:11.605159 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:11.738225 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:11.738108 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:11.833708 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:11.833668 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:12.542172 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:12.542138 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:12.572432 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:12.572393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:55:12.572649 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:12.572582 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 16:55:12.572649 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:12.572605 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd76bfb47-ktthd: secret "image-registry-tls" not found Jul 10 16:55:12.572717 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:12.572660 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls podName:4edf2bfe-ec0e-4469-8401-8f42470ade3b nodeName:}" failed. No retries permitted until 2026-07-10 16:55:28.57264487 +0000 UTC m=+68.698353258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls") pod "image-registry-7cd76bfb47-ktthd" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b") : secret "image-registry-tls" not found Jul 10 16:55:12.673776 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:12.673744 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:55:12.673893 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:12.673857 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:55:12.673929 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:12.673903 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls podName:f0d79cf9-d7b1-4e10-ab4e-59c351659657 nodeName:}" failed. No retries permitted until 2026-07-10 16:55:28.673889819 +0000 UTC m=+68.799598207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls") pod "dns-default-lmdzg" (UID: "f0d79cf9-d7b1-4e10-ab4e-59c351659657") : secret "dns-default-metrics-tls" not found Jul 10 16:55:13.141558 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:13.141518 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:13.339427 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:13.339372 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:13.777469 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:13.777437 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:13.938632 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:13.938588 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:14.329652 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:14.329599 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:14.738675 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:14.738631 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:14.935080 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:14.935046 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:15.035966 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:15.035882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:15.534350 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:15.534321 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:15.730988 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:15.730960 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:15.788521 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:15.788403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:15.943097 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:15.943062 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:16.041032 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:16.040960 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:16.140592 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:16.140555 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:16.365199 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:16.365169 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:16.964046 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:16.964010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:17.301317 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:17.301240 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:17.393844 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:17.393804 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:17.735718 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:17.735687 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:18.123334 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:18.123300 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:18.221554 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:18.221523 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:18.330867 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:18.330834 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:18.933762 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:18.933726 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:19.331554 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:19.331449 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:19.544027 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:19.543988 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:19.604568 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:19.604545 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:19.627796 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:19.627767 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:19.630229 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:19.630204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de9e01f8-0325-4387-a668-8dc75d3e13bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4frc\" (UID: \"de9e01f8-0325-4387-a668-8dc75d3e13bc\") " pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:19.929040 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:19.928952 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b4frc" Jul 10 16:55:19.938354 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:19.938328 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:20.060263 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:20.060232 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b4frc"] Jul 10 16:55:20.063793 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:20.063757 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9e01f8_0325_4387_a668_8dc75d3e13bc.slice/crio-49fe044d32175bcd2ff462cec7e030826297722162bbb78e5e976de7dbc74983 WatchSource:0}: Error finding container 49fe044d32175bcd2ff462cec7e030826297722162bbb78e5e976de7dbc74983: Status 404 returned error can't find the container with id 49fe044d32175bcd2ff462cec7e030826297722162bbb78e5e976de7dbc74983 Jul 10 16:55:20.736522 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:20.736451 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:20.748025 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:20.747985 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4frc" event={"ID":"de9e01f8-0325-4387-a668-8dc75d3e13bc","Type":"ContainerStarted","Data":"49fe044d32175bcd2ff462cec7e030826297722162bbb78e5e976de7dbc74983"} Jul 10 16:55:20.933688 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:20.933612 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:21.330568 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:21.330484 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:21.535435 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:21.535393 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:21.668294 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:21.668265 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lr5sl" Jul 10 16:55:21.732299 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:21.732265 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:24.760823 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:24.760787 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4frc" event={"ID":"de9e01f8-0325-4387-a668-8dc75d3e13bc","Type":"ContainerStarted","Data":"5ef10b841b24e4fafe7fdb86b139c9fa4775ffe5ae18674f20ab5f17b8fbc944"} Jul 10 16:55:24.761189 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:24.760851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4frc" event={"ID":"de9e01f8-0325-4387-a668-8dc75d3e13bc","Type":"ContainerStarted","Data":"debe3688e5ff6237faf5606beb28e128cfa1071c76e1abc28cbcddcb6cff956f"} Jul 10 16:55:26.084643 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.084594 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw64\" (UniqueName: \"kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64\") pod \"network-check-target-vr2l7\" (UID: \"afc23e34-857f-4c29-a381-cf87340044e7\") " pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:55:26.085114 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.084671 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs\") pod \"network-metrics-daemon-wg27k\" (UID: \"2e1b9d64-bc6b-4a12-86da-7cb47800704b\") " pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:55:26.087673 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.087648 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 16:55:26.088363 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.088346 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 16:55:26.097395 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.097365 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 16:55:26.097918 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.097892 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1b9d64-bc6b-4a12-86da-7cb47800704b-metrics-certs\") pod \"network-metrics-daemon-wg27k\" (UID: \"2e1b9d64-bc6b-4a12-86da-7cb47800704b\") " pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:55:26.108105 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.108077 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqw64\" (UniqueName: \"kubernetes.io/projected/afc23e34-857f-4c29-a381-cf87340044e7-kube-api-access-lqw64\") pod \"network-check-target-vr2l7\" (UID: \"afc23e34-857f-4c29-a381-cf87340044e7\") " pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:55:26.185312 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.185278 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:55:26.188098 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.188077 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 16:55:26.199119 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.199092 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a54dfc8-2f7b-4e80-9f23-03c128feeb24-original-pull-secret\") pod \"global-pull-secret-syncer-xdv2b\" (UID: \"2a54dfc8-2f7b-4e80-9f23-03c128feeb24\") " pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:55:26.203163 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.202949 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdv2b" Jul 10 16:55:26.210126 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.210107 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jn8d4\"" Jul 10 16:55:26.218298 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.218272 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:55:26.307462 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.307426 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdmq6\"" Jul 10 16:55:26.315736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:26.315701 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wg27k" Jul 10 16:55:27.234753 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:27.234719 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wg27k"] Jul 10 16:55:27.243372 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:27.243338 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1b9d64_bc6b_4a12_86da_7cb47800704b.slice/crio-cbf2b0856a27d328dc23fc73fff547faf5f8af8b3de93bd8470afd8002292273 WatchSource:0}: Error finding container cbf2b0856a27d328dc23fc73fff547faf5f8af8b3de93bd8470afd8002292273: Status 404 returned error can't find the container with id cbf2b0856a27d328dc23fc73fff547faf5f8af8b3de93bd8470afd8002292273 Jul 10 16:55:27.250947 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:27.250917 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vr2l7"] Jul 10 16:55:27.253722 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:27.253693 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc23e34_857f_4c29_a381_cf87340044e7.slice/crio-5652fd0ec77cdc314a01e92ba0b51fc343e7db47770d039f12ab3023e43548c1 WatchSource:0}: Error finding container 5652fd0ec77cdc314a01e92ba0b51fc343e7db47770d039f12ab3023e43548c1: Status 404 returned error can't find the container with id 5652fd0ec77cdc314a01e92ba0b51fc343e7db47770d039f12ab3023e43548c1 Jul 10 16:55:27.272185 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:27.272141 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xdv2b"] Jul 10 16:55:27.274875 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:27.274852 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a54dfc8_2f7b_4e80_9f23_03c128feeb24.slice/crio-9f4766b9b7d9c191f86aadaff7680d1ba207a55cab0c39235291acee786c4fd5 WatchSource:0}: Error finding container 9f4766b9b7d9c191f86aadaff7680d1ba207a55cab0c39235291acee786c4fd5: Status 404 returned error can't find the container with id 9f4766b9b7d9c191f86aadaff7680d1ba207a55cab0c39235291acee786c4fd5 Jul 10 16:55:27.774802 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:27.774760 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4frc" event={"ID":"de9e01f8-0325-4387-a668-8dc75d3e13bc","Type":"ContainerStarted","Data":"67b056cc0b6c5a7526c3391c0c6ac8af16bf7c48d05b7186e34ae352c7437906"} Jul 10 16:55:27.776481 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:27.776448 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xdv2b" event={"ID":"2a54dfc8-2f7b-4e80-9f23-03c128feeb24","Type":"ContainerStarted","Data":"9f4766b9b7d9c191f86aadaff7680d1ba207a55cab0c39235291acee786c4fd5"} Jul 10 16:55:27.777758 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:27.777732 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wg27k" event={"ID":"2e1b9d64-bc6b-4a12-86da-7cb47800704b","Type":"ContainerStarted","Data":"cbf2b0856a27d328dc23fc73fff547faf5f8af8b3de93bd8470afd8002292273"} Jul 10 16:55:27.778889 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:27.778867 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vr2l7" event={"ID":"afc23e34-857f-4c29-a381-cf87340044e7","Type":"ContainerStarted","Data":"5652fd0ec77cdc314a01e92ba0b51fc343e7db47770d039f12ab3023e43548c1"} Jul 10 16:55:28.605401 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:28.605340 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:55:28.608675 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:28.608645 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls\") pod \"image-registry-7cd76bfb47-ktthd\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:55:28.706039 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:28.705994 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:55:28.709185 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:28.709153 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0d79cf9-d7b1-4e10-ab4e-59c351659657-metrics-tls\") pod \"dns-default-lmdzg\" (UID: \"f0d79cf9-d7b1-4e10-ab4e-59c351659657\") " pod="openshift-dns/dns-default-lmdzg" Jul 10 16:55:28.784062 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:28.783977 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wg27k" event={"ID":"2e1b9d64-bc6b-4a12-86da-7cb47800704b","Type":"ContainerStarted","Data":"44cebc1185489817996d0e08433244243fec15bb5665567d927ca9762ab7bd0e"} Jul 10 16:55:28.784062 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:28.784033 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wg27k" event={"ID":"2e1b9d64-bc6b-4a12-86da-7cb47800704b","Type":"ContainerStarted","Data":"d20db2a1d5de96e4fe40669fd053fe53ee309e2538f7076e42caf9ad63f9fdf3"} Jul 10 16:55:28.801411 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:28.801353 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wg27k" podStartSLOduration=67.814686169 podStartE2EDuration="1m8.801336686s" totalImagesPullingTime="986.650517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:28.800431014 +0000 UTC m=+68.926139443" watchObservedRunningTime="2026-07-10 16:55:28.801336686 +0000 UTC m=+68.927045114" Jul 10 16:55:28.801843 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:28.801802 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b4frc" podStartSLOduration=18.848531067 podStartE2EDuration="25.801792976s" totalImagesPullingTime="6.953261909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:55:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:27.808297756 +0000 UTC m=+67.934006167" watchObservedRunningTime="2026-07-10 16:55:28.801792976 +0000 UTC m=+68.927501386" Jul 10 16:55:28.882227 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:28.882191 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-phvw6\"" Jul 10 16:55:28.890507 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:28.890469 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:55:28.894729 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:28.894697 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ccqfl\"" Jul 10 16:55:28.902836 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:28.902771 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lmdzg" Jul 10 16:55:29.076106 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.075969 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cd76bfb47-ktthd"] Jul 10 16:55:29.078445 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:29.078413 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4edf2bfe_ec0e_4469_8401_8f42470ade3b.slice/crio-cbcfb9936e226562911b2cd11f7a35436081af95829326c60404c8ca34cc2783 WatchSource:0}: Error finding container cbcfb9936e226562911b2cd11f7a35436081af95829326c60404c8ca34cc2783: Status 404 returned error can't find the container with id cbcfb9936e226562911b2cd11f7a35436081af95829326c60404c8ca34cc2783 Jul 10 16:55:29.094336 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.094313 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lmdzg"] Jul 10 16:55:29.095876 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:29.095843 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d79cf9_d7b1_4e10_ab4e_59c351659657.slice/crio-676487f0b7b25e237028eaca05b5b9283190e1640e5bf39ff2aaf79d397d1f2e WatchSource:0}: Error finding container 676487f0b7b25e237028eaca05b5b9283190e1640e5bf39ff2aaf79d397d1f2e: Status 404 returned error can't find the container with id 676487f0b7b25e237028eaca05b5b9283190e1640e5bf39ff2aaf79d397d1f2e Jul 10 16:55:29.678161 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.677258 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf"] Jul 10 16:55:29.696510 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.696155 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf" Jul 10 16:55:29.699837 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.697176 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf"] Jul 10 16:55:29.699837 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.699336 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cd76bfb47-ktthd"] Jul 10 16:55:29.700344 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.700315 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 16:55:29.700734 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.700715 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 10 16:55:29.702145 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.702123 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 16:55:29.702844 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.702658 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7tzdm\"" Jul 10 16:55:29.705166 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.705143 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 16:55:29.772977 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.772932 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78d8d8b7f5-btncf"] Jul 10 16:55:29.787589 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.787548 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:29.790570 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.790540 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:29.793263 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.793191 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78d8d8b7f5-btncf"] Jul 10 16:55:29.796113 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.795838 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" event={"ID":"4edf2bfe-ec0e-4469-8401-8f42470ade3b","Type":"ContainerStarted","Data":"c37fa068ac1faae78ae8a1f6b4eacf44bc7d5be74c75c30588b8421dd29374fc"} Jul 10 16:55:29.796113 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.795891 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" event={"ID":"4edf2bfe-ec0e-4469-8401-8f42470ade3b","Type":"ContainerStarted","Data":"cbcfb9936e226562911b2cd11f7a35436081af95829326c60404c8ca34cc2783"} Jul 10 16:55:29.796333 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.796312 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:55:29.802356 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.802326 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmdzg" event={"ID":"f0d79cf9-d7b1-4e10-ab4e-59c351659657","Type":"ContainerStarted","Data":"676487f0b7b25e237028eaca05b5b9283190e1640e5bf39ff2aaf79d397d1f2e"} Jul 10 16:55:29.815587 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.815554 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sqzl\" (UniqueName: \"kubernetes.io/projected/e6b182b5-5014-45e7-a180-68be9ca46a90-kube-api-access-8sqzl\") pod \"managed-serviceaccount-addon-agent-bcd57567b-6hzcf\" (UID: \"e6b182b5-5014-45e7-a180-68be9ca46a90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf" Jul 10 16:55:29.815762 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.815620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6b182b5-5014-45e7-a180-68be9ca46a90-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bcd57567b-6hzcf\" (UID: \"e6b182b5-5014-45e7-a180-68be9ca46a90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf" Jul 10 16:55:29.845227 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.845167 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" podStartSLOduration=49.845143308 podStartE2EDuration="49.845143308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:29.84491044 +0000 UTC m=+69.970618851" watchObservedRunningTime="2026-07-10 16:55:29.845143308 +0000 UTC m=+69.970851720" Jul 10 16:55:29.891772 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.891734 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:29.916821 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.916786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6b182b5-5014-45e7-a180-68be9ca46a90-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bcd57567b-6hzcf\" (UID: \"e6b182b5-5014-45e7-a180-68be9ca46a90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf" Jul 10 16:55:29.917012 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.916837 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrds\" (UniqueName: \"kubernetes.io/projected/9281e2bc-2561-4859-8421-26a5a3579ae9-kube-api-access-4lrds\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:29.917012 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.916882 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9281e2bc-2561-4859-8421-26a5a3579ae9-ca-trust-extracted\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:29.917012 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.916949 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9281e2bc-2561-4859-8421-26a5a3579ae9-installation-pull-secrets\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:29.917012 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.916983 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9281e2bc-2561-4859-8421-26a5a3579ae9-image-registry-private-configuration\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:29.917275 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.917018 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9281e2bc-2561-4859-8421-26a5a3579ae9-trusted-ca\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:29.917275 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.917148 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9281e2bc-2561-4859-8421-26a5a3579ae9-registry-certificates\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:29.917275 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.917180 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9281e2bc-2561-4859-8421-26a5a3579ae9-bound-sa-token\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:29.917275 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.917237 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sqzl\" (UniqueName: \"kubernetes.io/projected/e6b182b5-5014-45e7-a180-68be9ca46a90-kube-api-access-8sqzl\") pod \"managed-serviceaccount-addon-agent-bcd57567b-6hzcf\" (UID: \"e6b182b5-5014-45e7-a180-68be9ca46a90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf" Jul 10 16:55:29.917275 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.917259 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9281e2bc-2561-4859-8421-26a5a3579ae9-registry-tls\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:29.923047 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.923009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6b182b5-5014-45e7-a180-68be9ca46a90-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bcd57567b-6hzcf\" (UID: \"e6b182b5-5014-45e7-a180-68be9ca46a90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf" Jul 10 16:55:29.928370 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.928303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sqzl\" (UniqueName: \"kubernetes.io/projected/e6b182b5-5014-45e7-a180-68be9ca46a90-kube-api-access-8sqzl\") pod \"managed-serviceaccount-addon-agent-bcd57567b-6hzcf\" (UID: \"e6b182b5-5014-45e7-a180-68be9ca46a90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf" Jul 10 16:55:29.993991 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:29.993959 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:30.017852 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.017808 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9281e2bc-2561-4859-8421-26a5a3579ae9-registry-certificates\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.018031 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.017861 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9281e2bc-2561-4859-8421-26a5a3579ae9-bound-sa-token\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.018031 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.017894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9281e2bc-2561-4859-8421-26a5a3579ae9-registry-tls\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.018031 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.017930 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrds\" (UniqueName: \"kubernetes.io/projected/9281e2bc-2561-4859-8421-26a5a3579ae9-kube-api-access-4lrds\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.018031 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.017965 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9281e2bc-2561-4859-8421-26a5a3579ae9-ca-trust-extracted\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.018031 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.017997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9281e2bc-2561-4859-8421-26a5a3579ae9-installation-pull-secrets\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.018031 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.018030 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9281e2bc-2561-4859-8421-26a5a3579ae9-image-registry-private-configuration\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.018330 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.018062 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9281e2bc-2561-4859-8421-26a5a3579ae9-trusted-ca\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.019366 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.018756 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9281e2bc-2561-4859-8421-26a5a3579ae9-registry-certificates\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.019366 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.018854 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf" Jul 10 16:55:30.019366 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.018934 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9281e2bc-2561-4859-8421-26a5a3579ae9-ca-trust-extracted\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.019366 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.019028 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9281e2bc-2561-4859-8421-26a5a3579ae9-trusted-ca\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.022441 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.022080 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9281e2bc-2561-4859-8421-26a5a3579ae9-installation-pull-secrets\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.022441 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.022410 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9281e2bc-2561-4859-8421-26a5a3579ae9-registry-tls\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.024191 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.024161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9281e2bc-2561-4859-8421-26a5a3579ae9-image-registry-private-configuration\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.028870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.028821 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9281e2bc-2561-4859-8421-26a5a3579ae9-bound-sa-token\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.029949 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.029890 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrds\" (UniqueName: \"kubernetes.io/projected/9281e2bc-2561-4859-8421-26a5a3579ae9-kube-api-access-4lrds\") pod \"image-registry-78d8d8b7f5-btncf\" (UID: \"9281e2bc-2561-4859-8421-26a5a3579ae9\") " pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:30.096542 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.096444 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:30.105080 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:30.105052 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:31.076263 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.076228 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:31.177717 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.177679 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:31.306920 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.306886 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:31.475277 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.475243 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:31.585437 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.585398 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:31.720540 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.720505 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9dbgk"] Jul 10 16:55:31.722486 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.722461 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9dbgk" Jul 10 16:55:31.723604 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.723574 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:31.725034 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.725015 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 16:55:31.725893 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.725786 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ddscm\"" Jul 10 16:55:31.736159 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.736127 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9dbgk"] Jul 10 16:55:31.832721 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.832677 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/103306b3-c175-4203-9469-d180393e8f69-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-9dbgk\" (UID: \"103306b3-c175-4203-9469-d180393e8f69\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9dbgk" Jul 10 16:55:31.908333 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.908298 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:31.933382 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.933344 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/103306b3-c175-4203-9469-d180393e8f69-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-9dbgk\" (UID: \"103306b3-c175-4203-9469-d180393e8f69\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9dbgk" Jul 10 16:55:31.936330 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:31.936298 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/103306b3-c175-4203-9469-d180393e8f69-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-9dbgk\" (UID: \"103306b3-c175-4203-9469-d180393e8f69\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9dbgk" Jul 10 16:55:32.034857 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:32.034754 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9dbgk" Jul 10 16:55:33.167212 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.167063 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf"] Jul 10 16:55:33.170860 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:33.170691 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6b182b5_5014_45e7_a180_68be9ca46a90.slice/crio-dbd8976c151fe50473da5b2c22155f0b8b654ada3ec5bf373b1db36250039a02 WatchSource:0}: Error finding container dbd8976c151fe50473da5b2c22155f0b8b654ada3ec5bf373b1db36250039a02: Status 404 returned error can't find the container with id dbd8976c151fe50473da5b2c22155f0b8b654ada3ec5bf373b1db36250039a02 Jul 10 16:55:33.385465 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.385441 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9dbgk"] Jul 10 16:55:33.388639 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:33.388604 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103306b3_c175_4203_9469_d180393e8f69.slice/crio-a76a90ec360dc27daff361c49055de37fabe36d931468acfeb23cefa4562adf3 WatchSource:0}: Error finding container a76a90ec360dc27daff361c49055de37fabe36d931468acfeb23cefa4562adf3: Status 404 returned error can't find the container with id a76a90ec360dc27daff361c49055de37fabe36d931468acfeb23cefa4562adf3 Jul 10 16:55:33.388985 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.388954 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78d8d8b7f5-btncf"] Jul 10 16:55:33.391283 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:33.391263 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9281e2bc_2561_4859_8421_26a5a3579ae9.slice/crio-0e91d5bae02232552d8b71c7401df277c1753e79c4543b399f67a59051f13ae6 WatchSource:0}: Error finding container 0e91d5bae02232552d8b71c7401df277c1753e79c4543b399f67a59051f13ae6: Status 404 returned error can't find the container with id 0e91d5bae02232552d8b71c7401df277c1753e79c4543b399f67a59051f13ae6 Jul 10 16:55:33.817998 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.817959 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vr2l7" event={"ID":"afc23e34-857f-4c29-a381-cf87340044e7","Type":"ContainerStarted","Data":"8ddbc5f30caa9061cd231ce9b05e1a1b677d3d5147fcec5c25ed79ea4d0ba7be"} Jul 10 16:55:33.818180 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.818079 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:55:33.819436 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.819411 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" event={"ID":"9281e2bc-2561-4859-8421-26a5a3579ae9","Type":"ContainerStarted","Data":"834974ac9cfdfd3efe900d8c6d79feca4183fd1f88e3754f33db8719e864efe8"} Jul 10 16:55:33.819546 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.819450 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" event={"ID":"9281e2bc-2561-4859-8421-26a5a3579ae9","Type":"ContainerStarted","Data":"0e91d5bae02232552d8b71c7401df277c1753e79c4543b399f67a59051f13ae6"} Jul 10 16:55:33.819602 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.819543 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:33.820866 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.820841 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xdv2b" event={"ID":"2a54dfc8-2f7b-4e80-9f23-03c128feeb24","Type":"ContainerStarted","Data":"ef4e3ad7e28173bfc14defea397179a46de31887929660c1c4a176b4f09d8c37"} Jul 10 16:55:33.821839 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.821817 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9dbgk" event={"ID":"103306b3-c175-4203-9469-d180393e8f69","Type":"ContainerStarted","Data":"a76a90ec360dc27daff361c49055de37fabe36d931468acfeb23cefa4562adf3"} Jul 10 16:55:33.822919 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.822899 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf" event={"ID":"e6b182b5-5014-45e7-a180-68be9ca46a90","Type":"ContainerStarted","Data":"dbd8976c151fe50473da5b2c22155f0b8b654ada3ec5bf373b1db36250039a02"} Jul 10 16:55:33.824471 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.824452 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmdzg" event={"ID":"f0d79cf9-d7b1-4e10-ab4e-59c351659657","Type":"ContainerStarted","Data":"4f7b2436475a23edbfc551be5c91b8f34f814f1b1c477589e4eba5a16b6663ea"} Jul 10 16:55:33.824574 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.824525 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmdzg" event={"ID":"f0d79cf9-d7b1-4e10-ab4e-59c351659657","Type":"ContainerStarted","Data":"86242b0daeaf7cd870df34dd05d7feeec51c797fb51441c468848a55f6b9b003"} Jul 10 16:55:33.824574 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.824554 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lmdzg" Jul 10 16:55:33.834692 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.834649 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vr2l7" podStartSLOduration=68.09022185 podStartE2EDuration="1m13.834631608s" totalImagesPullingTime="5.744409758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:33.834441598 +0000 UTC m=+73.960150007" watchObservedRunningTime="2026-07-10 16:55:33.834631608 +0000 UTC m=+73.960340036" Jul 10 16:55:33.849638 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.849586 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xdv2b" podStartSLOduration=67.124959391 podStartE2EDuration="1m12.849569552s" totalImagesPullingTime="5.724610161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:33.848845369 +0000 UTC m=+73.974553783" watchObservedRunningTime="2026-07-10 16:55:33.849569552 +0000 UTC m=+73.975277939" Jul 10 16:55:33.866016 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.865962 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lmdzg" podStartSLOduration=33.960853502 podStartE2EDuration="37.865944396s" totalImagesPullingTime="3.905090894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:33.865347978 +0000 UTC m=+73.991056412" watchObservedRunningTime="2026-07-10 16:55:33.865944396 +0000 UTC m=+73.991652807" Jul 10 16:55:33.883622 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:33.883574 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" podStartSLOduration=4.883558853 podStartE2EDuration="4.883558853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:33.883450885 +0000 UTC m=+74.009159300" watchObservedRunningTime="2026-07-10 16:55:33.883558853 +0000 UTC m=+74.009267262" Jul 10 16:55:34.546090 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:34.546048 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:34.619657 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:34.619621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:34.830399 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:34.830301 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9dbgk" event={"ID":"103306b3-c175-4203-9469-d180393e8f69","Type":"ContainerStarted","Data":"5d8a9f73f9237aa89b20ee5bc1d8924e66ea587ad271a7aa25a251b8f75b999b"} Jul 10 16:55:34.849190 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:34.849135 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9dbgk" podStartSLOduration=2.56766305 podStartE2EDuration="3.84911759s" totalImagesPullingTime="1.28145454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:34.848161938 +0000 UTC m=+74.973870347" watchObservedRunningTime="2026-07-10 16:55:34.84911759 +0000 UTC m=+74.974826000" Jul 10 16:55:35.834306 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:35.834283 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9dbgk" Jul 10 16:55:35.839591 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:35.839569 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9dbgk" Jul 10 16:55:36.752432 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:36.752397 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:36.843402 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:36.843360 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf" event={"ID":"e6b182b5-5014-45e7-a180-68be9ca46a90","Type":"ContainerStarted","Data":"d8d866d45c2532b334ad20583edece3ca86c1f93774ef9bd5a404235777bb11d"} Jul 10 16:55:36.855189 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:36.855163 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:37.039891 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:37.039791 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:38.476668 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.476613 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bcd57567b-6hzcf" podStartSLOduration=6.854872727 podStartE2EDuration="9.476593106s" totalImagesPullingTime="2.621720379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:36.877130765 +0000 UTC m=+77.002839175" watchObservedRunningTime="2026-07-10 16:55:38.476593106 +0000 UTC m=+78.602301560" Jul 10 16:55:38.477526 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.477507 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-2xv2r"] Jul 10 16:55:38.480674 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.480659 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-2xv2r" Jul 10 16:55:38.483646 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.483623 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-pwzbc\"" Jul 10 16:55:38.483814 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.483797 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 16:55:38.485993 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.485972 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 16:55:38.492568 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.492475 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-2xv2r"] Jul 10 16:55:38.497727 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.497701 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:38.586765 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.586726 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lspv2\" (UniqueName: \"kubernetes.io/projected/4aad8b73-7fd5-4f1d-89f8-ec9bdb1afb4d-kube-api-access-lspv2\") pod \"downloads-68c9cbb497-2xv2r\" (UID: \"4aad8b73-7fd5-4f1d-89f8-ec9bdb1afb4d\") " pod="openshift-console/downloads-68c9cbb497-2xv2r" Jul 10 16:55:38.601419 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.601381 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:38.687312 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.687273 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lspv2\" (UniqueName: \"kubernetes.io/projected/4aad8b73-7fd5-4f1d-89f8-ec9bdb1afb4d-kube-api-access-lspv2\") pod \"downloads-68c9cbb497-2xv2r\" (UID: \"4aad8b73-7fd5-4f1d-89f8-ec9bdb1afb4d\") " pod="openshift-console/downloads-68c9cbb497-2xv2r" Jul 10 16:55:38.698236 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.698210 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lspv2\" (UniqueName: \"kubernetes.io/projected/4aad8b73-7fd5-4f1d-89f8-ec9bdb1afb4d-kube-api-access-lspv2\") pod \"downloads-68c9cbb497-2xv2r\" (UID: \"4aad8b73-7fd5-4f1d-89f8-ec9bdb1afb4d\") " pod="openshift-console/downloads-68c9cbb497-2xv2r" Jul 10 16:55:38.746521 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.746407 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:38.790045 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.790016 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-2xv2r" Jul 10 16:55:38.859061 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.858942 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:38.947001 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:38.946981 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-2xv2r"] Jul 10 16:55:38.948724 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:38.948700 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aad8b73_7fd5_4f1d_89f8_ec9bdb1afb4d.slice/crio-8bf4226d90f9a85f3d73f3698710dc4a1725b7e5c6ad46bbd24e069080bcec1c WatchSource:0}: Error finding container 8bf4226d90f9a85f3d73f3698710dc4a1725b7e5c6ad46bbd24e069080bcec1c: Status 404 returned error can't find the container with id 8bf4226d90f9a85f3d73f3698710dc4a1725b7e5c6ad46bbd24e069080bcec1c Jul 10 16:55:39.137363 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:39.137323 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:39.210197 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:39.210163 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:39.532995 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:39.532911 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:39.591803 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:39.591764 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:39.736790 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:39.736757 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:39.815664 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:39.815574 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:39.882157 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:39.882113 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-2xv2r" event={"ID":"4aad8b73-7fd5-4f1d-89f8-ec9bdb1afb4d","Type":"ContainerStarted","Data":"8bf4226d90f9a85f3d73f3698710dc4a1725b7e5c6ad46bbd24e069080bcec1c"} Jul 10 16:55:39.947020 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:39.946980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:40.173873 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:40.173836 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:40.272514 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:40.272462 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:40.371901 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:40.371868 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:40.432051 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:40.431961 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:40.933573 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:40.933537 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:41.151259 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.151232 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:41.172301 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.172266 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6"] Jul 10 16:55:41.175166 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.175149 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:41.178009 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.177980 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 10 16:55:41.178122 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.178041 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 16:55:41.178122 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.177980 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 16:55:41.179075 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.179055 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 16:55:41.179286 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.179266 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 10 16:55:41.179409 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.179302 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-t6dfh\"" Jul 10 16:55:41.192287 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.192261 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6"] Jul 10 16:55:41.199237 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.199199 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-jftrl"] Jul 10 16:55:41.202300 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.202278 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.205217 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.205167 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 10 16:55:41.205217 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.205188 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 10 16:55:41.205217 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.205210 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6t7cs\"" Jul 10 16:55:41.205445 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.205176 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 10 16:55:41.222438 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.222364 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-jftrl"] Jul 10 16:55:41.223901 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.223853 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-prsg2"] Jul 10 16:55:41.227157 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.227073 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.229662 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.229637 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 16:55:41.229849 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.229644 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 16:55:41.229966 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.229876 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 16:55:41.231331 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.231308 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kbv2t\"" Jul 10 16:55:41.254116 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.254074 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:41.308061 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308017 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.308061 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308060 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2xkw\" (UniqueName: \"kubernetes.io/projected/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-kube-api-access-g2xkw\") pod \"openshift-state-metrics-84bd68b896-z75w6\" (UID: \"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:41.308244 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308086 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-accelerators-collector-config\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.308244 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308105 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c71664a0-87cd-4b6b-96b6-88aff6ced117-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.308244 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308130 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-z75w6\" (UID: \"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:41.308244 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308150 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-metrics-client-ca\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.308244 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308167 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-sys\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.308244 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308182 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.308244 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308196 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-root\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.308244 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308229 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps95g\" (UniqueName: \"kubernetes.io/projected/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-api-access-ps95g\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.308244 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308245 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-z75w6\" (UID: \"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:41.308520 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308268 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-z75w6\" (UID: \"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:41.308520 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308284 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-textfile\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.308520 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308300 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-wtmp\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.308520 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308316 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.308520 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308335 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.308520 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308356 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-tls\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.308520 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308373 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4bg\" (UniqueName: \"kubernetes.io/projected/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-kube-api-access-bx4bg\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.308520 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.308387 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c71664a0-87cd-4b6b-96b6-88aff6ced117-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.348853 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.348782 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:41.409478 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409443 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-z75w6\" (UID: \"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:41.409695 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409488 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-metrics-client-ca\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.409695 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409540 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-sys\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.409695 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.409695 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409587 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-root\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.409695 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409632 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps95g\" (UniqueName: \"kubernetes.io/projected/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-api-access-ps95g\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.409695 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409658 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-z75w6\" (UID: \"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:41.409695 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409687 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-z75w6\" (UID: \"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:41.410038 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-textfile\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.410038 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409738 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-wtmp\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.410038 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409766 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.410038 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409793 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.410038 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-tls\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.410038 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409854 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4bg\" (UniqueName: \"kubernetes.io/projected/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-kube-api-access-bx4bg\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.410038 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c71664a0-87cd-4b6b-96b6-88aff6ced117-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.410038 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.410038 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409951 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2xkw\" (UniqueName: \"kubernetes.io/projected/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-kube-api-access-g2xkw\") pod \"openshift-state-metrics-84bd68b896-z75w6\" (UID: \"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:41.410038 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.409984 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-accelerators-collector-config\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.410038 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.410009 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c71664a0-87cd-4b6b-96b6-88aff6ced117-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.410558 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.410440 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c71664a0-87cd-4b6b-96b6-88aff6ced117-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.410649 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.410620 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-root\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.410713 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.410684 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-wtmp\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.410763 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:41.410704 2559 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 10 16:55:41.410813 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:41.410778 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-openshift-state-metrics-tls podName:e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0 nodeName:}" failed. No retries permitted until 2026-07-10 16:55:41.910756348 +0000 UTC m=+82.036464743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-z75w6" (UID: "e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0") : secret "openshift-state-metrics-tls" not found Jul 10 16:55:41.412442 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.411152 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-z75w6\" (UID: \"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:41.412442 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.411233 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-metrics-client-ca\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.412442 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.411289 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-sys\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.412442 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.411972 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c71664a0-87cd-4b6b-96b6-88aff6ced117-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.412442 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.411980 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-textfile\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.412442 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.412043 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.412442 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:41.412085 2559 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 10 16:55:41.412442 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:41.412137 2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 16:55:41.412442 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:41.412149 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-state-metrics-tls podName:c71664a0-87cd-4b6b-96b6-88aff6ced117 nodeName:}" failed. No retries permitted until 2026-07-10 16:55:41.912132276 +0000 UTC m=+82.037840670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-jftrl" (UID: "c71664a0-87cd-4b6b-96b6-88aff6ced117") : secret "kube-state-metrics-tls" not found Jul 10 16:55:41.412442 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:41.412192 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-tls podName:8e1bdb18-9cb3-4a14-9368-35be0c45a25d nodeName:}" failed. No retries permitted until 2026-07-10 16:55:41.912176496 +0000 UTC m=+82.037884887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-tls") pod "node-exporter-prsg2" (UID: "8e1bdb18-9cb3-4a14-9368-35be0c45a25d") : secret "node-exporter-tls" not found Jul 10 16:55:41.412442 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.412401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-accelerators-collector-config\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.416272 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.415737 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.416908 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.416651 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.416908 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.416832 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-z75w6\" (UID: \"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:41.428678 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.425124 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps95g\" (UniqueName: \"kubernetes.io/projected/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-api-access-ps95g\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.428678 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.428103 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2xkw\" (UniqueName: \"kubernetes.io/projected/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-kube-api-access-g2xkw\") pod \"openshift-state-metrics-84bd68b896-z75w6\" (UID: \"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:41.428678 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.428438 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4bg\" (UniqueName: \"kubernetes.io/projected/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-kube-api-access-bx4bg\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.480744 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.480658 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:41.595034 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.594990 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:41.703014 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.702980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:41.804290 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.804210 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:41.914640 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.914598 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-z75w6\" (UID: \"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:41.914818 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.914666 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.914818 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.914700 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-tls\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.917748 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.917720 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c71664a0-87cd-4b6b-96b6-88aff6ced117-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-jftrl\" (UID: \"c71664a0-87cd-4b6b-96b6-88aff6ced117\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:41.917907 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.917884 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e1bdb18-9cb3-4a14-9368-35be0c45a25d-node-exporter-tls\") pod \"node-exporter-prsg2\" (UID: \"8e1bdb18-9cb3-4a14-9368-35be0c45a25d\") " pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:41.918001 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:41.917907 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-z75w6\" (UID: \"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:42.086870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.086787 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" Jul 10 16:55:42.115867 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.115662 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" Jul 10 16:55:42.139737 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.139440 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-prsg2" Jul 10 16:55:42.152726 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.152658 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:42.254409 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.254364 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:42.296391 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.296347 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6"] Jul 10 16:55:42.301763 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:42.301730 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode151fc47_6131_4a64_b0ff_6fd1f0fc4ee0.slice/crio-681dff7b3c5bbf8912dabffacd09fa0f1f8d0887ef78e9ba62f56d45e68f9b76 WatchSource:0}: Error finding container 681dff7b3c5bbf8912dabffacd09fa0f1f8d0887ef78e9ba62f56d45e68f9b76: Status 404 returned error can't find the container with id 681dff7b3c5bbf8912dabffacd09fa0f1f8d0887ef78e9ba62f56d45e68f9b76 Jul 10 16:55:42.324639 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.324308 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-jftrl"] Jul 10 16:55:42.334693 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:42.334667 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71664a0_87cd_4b6b_96b6_88aff6ced117.slice/crio-0fd50ba308fbef931a93b6391d65469bb8fe1b0b09f367ea9f91112b02249fcc WatchSource:0}: Error finding container 0fd50ba308fbef931a93b6391d65469bb8fe1b0b09f367ea9f91112b02249fcc: Status 404 returned error can't find the container with id 0fd50ba308fbef931a93b6391d65469bb8fe1b0b09f367ea9f91112b02249fcc Jul 10 16:55:42.373984 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.373954 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:42.574749 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.574713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:42.895644 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.895605 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prsg2" event={"ID":"8e1bdb18-9cb3-4a14-9368-35be0c45a25d","Type":"ContainerStarted","Data":"7d83032360b03752595c2c50e367c226f6516fa30f0b811c6f1b43c4aded7dc2"} Jul 10 16:55:42.897201 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.897168 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" event={"ID":"c71664a0-87cd-4b6b-96b6-88aff6ced117","Type":"ContainerStarted","Data":"0fd50ba308fbef931a93b6391d65469bb8fe1b0b09f367ea9f91112b02249fcc"} Jul 10 16:55:42.899535 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.899486 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" event={"ID":"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0","Type":"ContainerStarted","Data":"15c9adaaeb6001fd16ce86e5b1fc8374a19ea8d7fc0c3d4c7e1f5192f623fe64"} Jul 10 16:55:42.899675 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.899549 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" event={"ID":"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0","Type":"ContainerStarted","Data":"bbf5c9a4b2e6e2d3a570ced2c1f1e60c9d53392c31325391c05031638c9623bd"} Jul 10 16:55:42.899675 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:42.899568 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" event={"ID":"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0","Type":"ContainerStarted","Data":"681dff7b3c5bbf8912dabffacd09fa0f1f8d0887ef78e9ba62f56d45e68f9b76"} Jul 10 16:55:43.160901 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.160818 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b97567679-7mgmg"] Jul 10 16:55:43.167586 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.166408 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.171350 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.170824 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fgknc\"" Jul 10 16:55:43.171350 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.170876 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 10 16:55:43.171350 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.170830 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 10 16:55:43.171350 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.171068 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 10 16:55:43.171715 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.171638 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 10 16:55:43.172182 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.171966 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2t98lfhlr24kv\"" Jul 10 16:55:43.172182 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.171966 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 10 16:55:43.176977 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.176553 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b97567679-7mgmg"] Jul 10 16:55:43.225564 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.225524 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.225726 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.225587 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.225726 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.225621 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/091039e8-bb5e-4889-8319-7c416fe7a79d-metrics-client-ca\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.225726 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.225654 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.225726 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.225694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-tls\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.225935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.225716 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-grpc-tls\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.225935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.225770 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.225935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.225798 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k4g5\" (UniqueName: \"kubernetes.io/projected/091039e8-bb5e-4889-8319-7c416fe7a79d-kube-api-access-4k4g5\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.327168 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.326248 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.327168 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.326309 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.327168 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.326343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/091039e8-bb5e-4889-8319-7c416fe7a79d-metrics-client-ca\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.327168 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.326381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.327168 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.326423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-tls\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.327168 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.326446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-grpc-tls\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.327168 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.326486 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.327168 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.326537 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k4g5\" (UniqueName: \"kubernetes.io/projected/091039e8-bb5e-4889-8319-7c416fe7a79d-kube-api-access-4k4g5\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.328454 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.328422 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/091039e8-bb5e-4889-8319-7c416fe7a79d-metrics-client-ca\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.330731 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.330706 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-grpc-tls\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.331823 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.331801 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.331823 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.331816 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-tls\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.334222 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.334192 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.335434 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.335412 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.336121 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.336099 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k4g5\" (UniqueName: \"kubernetes.io/projected/091039e8-bb5e-4889-8319-7c416fe7a79d-kube-api-access-4k4g5\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.337671 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.337609 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/091039e8-bb5e-4889-8319-7c416fe7a79d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b97567679-7mgmg\" (UID: \"091039e8-bb5e-4889-8319-7c416fe7a79d\") " pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.484563 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.484451 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:43.834433 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.833541 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lmdzg" Jul 10 16:55:43.907092 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.907057 2559 generic.go:356] "Generic (PLEG): container finished" podID="8e1bdb18-9cb3-4a14-9368-35be0c45a25d" containerID="1ef3d5bd21467a06345147712f8a3259247daf88172611fd8150e623222df94a" exitCode=0 Jul 10 16:55:43.907251 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:43.907163 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prsg2" event={"ID":"8e1bdb18-9cb3-4a14-9368-35be0c45a25d","Type":"ContainerDied","Data":"1ef3d5bd21467a06345147712f8a3259247daf88172611fd8150e623222df94a"} Jul 10 16:55:44.114973 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.114936 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:44.183922 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.183883 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:44.277256 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.277124 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b97567679-7mgmg"] Jul 10 16:55:44.282405 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:44.282373 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091039e8_bb5e_4889_8319_7c416fe7a79d.slice/crio-0f4b7329b9e3123722dfa983ec02a0a014b0ce7f0c027890ac55816f1e582ede WatchSource:0}: Error finding container 0f4b7329b9e3123722dfa983ec02a0a014b0ce7f0c027890ac55816f1e582ede: Status 404 returned error can't find the container with id 0f4b7329b9e3123722dfa983ec02a0a014b0ce7f0c027890ac55816f1e582ede Jul 10 16:55:44.456325 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.456293 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:44.680775 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.680691 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:44.916032 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.915993 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" event={"ID":"e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0","Type":"ContainerStarted","Data":"b8ee20adec3d9c0caf7c2b69322772b6818b7931e5df8af44e99d90ed3bd579f"} Jul 10 16:55:44.918437 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.918339 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" event={"ID":"091039e8-bb5e-4889-8319-7c416fe7a79d","Type":"ContainerStarted","Data":"0f4b7329b9e3123722dfa983ec02a0a014b0ce7f0c027890ac55816f1e582ede"} Jul 10 16:55:44.921222 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.921194 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prsg2" event={"ID":"8e1bdb18-9cb3-4a14-9368-35be0c45a25d","Type":"ContainerStarted","Data":"41ed0ddaf92a4408645315960b08dd0174f3557f9f04103a8e282aff835aa9b4"} Jul 10 16:55:44.921339 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.921241 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prsg2" event={"ID":"8e1bdb18-9cb3-4a14-9368-35be0c45a25d","Type":"ContainerStarted","Data":"a71d1491e741b5ea2a7e546b5e86cecc075827e5d106c58433e4506ad13b6685"} Jul 10 16:55:44.925684 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.925658 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" event={"ID":"c71664a0-87cd-4b6b-96b6-88aff6ced117","Type":"ContainerStarted","Data":"2a9a4aab0482bc7062bb5bcb476cf0be8e55a04673da4b0fc1144783d3267242"} Jul 10 16:55:44.925799 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.925716 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" event={"ID":"c71664a0-87cd-4b6b-96b6-88aff6ced117","Type":"ContainerStarted","Data":"ff3c6adcc87b202cda112f7762fac9fae8b46f9abd39a312f5bc2370665a4bf9"} Jul 10 16:55:44.925799 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.925757 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" event={"ID":"c71664a0-87cd-4b6b-96b6-88aff6ced117","Type":"ContainerStarted","Data":"5f4842c4acbb1c67bbdd642037ed67ea3a45331fa8ce87a87ff2b6c307a608e1"} Jul 10 16:55:44.936251 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.936149 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z75w6" podStartSLOduration=2.315871579 podStartE2EDuration="3.936128802s" totalImagesPullingTime="1.620257223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:44.935371003 +0000 UTC m=+85.061079414" watchObservedRunningTime="2026-07-10 16:55:44.936128802 +0000 UTC m=+85.061837214" Jul 10 16:55:44.955211 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.955154 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-jftrl" podStartSLOduration=2.205200878 podStartE2EDuration="3.955141928s" totalImagesPullingTime="1.74994105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:44.954788815 +0000 UTC m=+85.080497226" watchObservedRunningTime="2026-07-10 16:55:44.955141928 +0000 UTC m=+85.080850339" Jul 10 16:55:44.974542 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:44.974452 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-prsg2" podStartSLOduration=2.727202343 podStartE2EDuration="3.974429333s" totalImagesPullingTime="1.152206866s" totalInitContainerRuntime="95.020124ms" isStatefulPod=true podCreationTimestamp="2026-07-10 16:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:44.973486816 +0000 UTC m=+85.099195221" watchObservedRunningTime="2026-07-10 16:55:44.974429333 +0000 UTC m=+85.100137748" Jul 10 16:55:45.946425 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:45.946384 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck"] Jul 10 16:55:45.961459 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:45.961009 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:45.963485 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:45.962487 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck"] Jul 10 16:55:45.963485 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:45.962632 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck" Jul 10 16:55:45.965408 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:45.965380 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 10 16:55:45.965547 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:45.965388 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-l2bg8\"" Jul 10 16:55:46.021316 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:46.021277 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:46.059305 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:46.058459 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a52675af-70bc-428c-9efc-ee6308b051f4-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-mcnck\" (UID: \"a52675af-70bc-428c-9efc-ee6308b051f4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck" Jul 10 16:55:46.059305 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:46.058697 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x628k\" (UniqueName: \"kubernetes.io/projected/a52675af-70bc-428c-9efc-ee6308b051f4-kube-api-access-x628k\") pod \"monitoring-plugin-5f68df7f58-mcnck\" (UID: \"a52675af-70bc-428c-9efc-ee6308b051f4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck" Jul 10 16:55:46.160068 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:46.160033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a52675af-70bc-428c-9efc-ee6308b051f4-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-mcnck\" (UID: \"a52675af-70bc-428c-9efc-ee6308b051f4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck" Jul 10 16:55:46.160255 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:46.160118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x628k\" (UniqueName: \"kubernetes.io/projected/a52675af-70bc-428c-9efc-ee6308b051f4-kube-api-access-x628k\") pod \"monitoring-plugin-5f68df7f58-mcnck\" (UID: \"a52675af-70bc-428c-9efc-ee6308b051f4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck" Jul 10 16:55:46.161777 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:46.161674 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:46.163351 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:46.163298 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a52675af-70bc-428c-9efc-ee6308b051f4-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-mcnck\" (UID: \"a52675af-70bc-428c-9efc-ee6308b051f4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck" Jul 10 16:55:46.172220 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:46.172194 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x628k\" (UniqueName: \"kubernetes.io/projected/a52675af-70bc-428c-9efc-ee6308b051f4-kube-api-access-x628k\") pod \"monitoring-plugin-5f68df7f58-mcnck\" (UID: \"a52675af-70bc-428c-9efc-ee6308b051f4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck" Jul 10 16:55:46.280466 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:46.280370 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck" Jul 10 16:55:46.937623 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:46.937587 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" event={"ID":"091039e8-bb5e-4889-8319-7c416fe7a79d","Type":"ContainerStarted","Data":"42bbc18d86abd1b2790ec1b706969f63f34c2725ad71076e94d7901c1fe688cf"} Jul 10 16:55:46.954387 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:46.954311 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck"] Jul 10 16:55:46.960771 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:55:46.960732 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52675af_70bc_428c_9efc_ee6308b051f4.slice/crio-41c06fabb3a3a67551c9bd1ac3fa38236e78309e8c856e8acee5fa755dfefe85 WatchSource:0}: Error finding container 41c06fabb3a3a67551c9bd1ac3fa38236e78309e8c856e8acee5fa755dfefe85: Status 404 returned error can't find the container with id 41c06fabb3a3a67551c9bd1ac3fa38236e78309e8c856e8acee5fa755dfefe85 Jul 10 16:55:47.733415 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:47.733377 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:47.947274 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:47.947236 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck" event={"ID":"a52675af-70bc-428c-9efc-ee6308b051f4","Type":"ContainerStarted","Data":"41c06fabb3a3a67551c9bd1ac3fa38236e78309e8c856e8acee5fa755dfefe85"} Jul 10 16:55:47.949474 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:47.949430 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" event={"ID":"091039e8-bb5e-4889-8319-7c416fe7a79d","Type":"ContainerStarted","Data":"215c2a6b888de4ba5137c14b7253deb65b336e7df5b2645da5a56cefcf43c3a8"} Jul 10 16:55:47.949623 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:47.949523 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" event={"ID":"091039e8-bb5e-4889-8319-7c416fe7a79d","Type":"ContainerStarted","Data":"34e77607c0902e2bd8051f81ff63a712f65961f0c8daf2a09381e79447b2cedf"} Jul 10 16:55:48.251704 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:48.251669 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:48.342481 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:48.342449 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:48.955645 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:48.955597 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck" event={"ID":"a52675af-70bc-428c-9efc-ee6308b051f4","Type":"ContainerStarted","Data":"29b72058c7fb5c4e404032c699cee6845421cf909949f22df31ccde3e4826492"} Jul 10 16:55:48.955878 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:48.955706 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck" Jul 10 16:55:48.959211 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:48.959152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" event={"ID":"091039e8-bb5e-4889-8319-7c416fe7a79d","Type":"ContainerStarted","Data":"cf25c9f429717e60e3fc2169ccc91158686bd57f0c4648f692ae3de6d56bd0a9"} Jul 10 16:55:48.959381 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:48.959227 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" event={"ID":"091039e8-bb5e-4889-8319-7c416fe7a79d","Type":"ContainerStarted","Data":"69ede32a330b201b5049d2124ceef9ab78299f4898dfea061b09f6c38365cef2"} Jul 10 16:55:48.959381 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:48.959266 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" event={"ID":"091039e8-bb5e-4889-8319-7c416fe7a79d","Type":"ContainerStarted","Data":"bae6040b2c814257182152e851703e4e906e520d33c2cc279019c468fe796289"} Jul 10 16:55:48.959381 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:48.959359 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:48.962070 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:48.962047 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck" Jul 10 16:55:48.972027 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:48.971977 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-mcnck" podStartSLOduration=2.580534193 podStartE2EDuration="3.971957867s" totalImagesPullingTime="1.391423674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:48.970975118 +0000 UTC m=+89.096683530" watchObservedRunningTime="2026-07-10 16:55:48.971957867 +0000 UTC m=+89.097666278" Jul 10 16:55:48.994633 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:48.994551 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" podStartSLOduration=2.590659315 podStartE2EDuration="5.994531545s" totalImagesPullingTime="3.40387223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:48.993782452 +0000 UTC m=+89.119490874" watchObservedRunningTime="2026-07-10 16:55:48.994531545 +0000 UTC m=+89.120239958" Jul 10 16:55:49.544154 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:49.544118 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:49.606849 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:49.606806 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:50.363044 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:50.363003 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:50.465274 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:50.465236 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:50.813424 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:50.813320 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:55:51.125128 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:51.125098 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:51.225171 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:51.225141 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:54.468283 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:54.468244 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:54.568761 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:54.568723 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:54.839783 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:54.839074 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78d8d8b7f5-btncf" Jul 10 16:55:54.971878 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:54.971843 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b97567679-7mgmg" Jul 10 16:55:54.982456 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:54.982419 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:55.136885 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:55.136747 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:55.233862 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:55.233832 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:55.831245 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:55.831194 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" podUID="4edf2bfe-ec0e-4469-8401-8f42470ade3b" containerName="registry" containerID="cri-o://c37fa068ac1faae78ae8a1f6b4eacf44bc7d5be74c75c30588b8421dd29374fc" gracePeriod=30 Jul 10 16:55:56.118725 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.118692 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:56.178338 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.178300 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:56.820046 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.820005 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:55:56.955407 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.955378 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-bound-sa-token\") pod \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " Jul 10 16:55:56.955746 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.955424 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4edf2bfe-ec0e-4469-8401-8f42470ade3b-ca-trust-extracted\") pod \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " Jul 10 16:55:56.955746 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.955603 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4edf2bfe-ec0e-4469-8401-8f42470ade3b-image-registry-private-configuration\") pod \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " Jul 10 16:55:56.955746 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.955643 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4edf2bfe-ec0e-4469-8401-8f42470ade3b-installation-pull-secrets\") pod \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " Jul 10 16:55:56.955746 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.955684 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk6lv\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-kube-api-access-vk6lv\") pod \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " Jul 10 16:55:56.955746 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.955713 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls\") pod \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " Jul 10 16:55:56.955746 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.955738 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4edf2bfe-ec0e-4469-8401-8f42470ade3b-trusted-ca\") pod \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " Jul 10 16:55:56.956622 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.956127 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-certificates\") pod \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\" (UID: \"4edf2bfe-ec0e-4469-8401-8f42470ade3b\") " Jul 10 16:55:56.956622 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.956169 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4edf2bfe-ec0e-4469-8401-8f42470ade3b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4edf2bfe-ec0e-4469-8401-8f42470ade3b" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:55:56.956622 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.956582 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4edf2bfe-ec0e-4469-8401-8f42470ade3b-trusted-ca\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:55:56.956867 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.956838 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4edf2bfe-ec0e-4469-8401-8f42470ade3b" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:55:56.958302 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.958250 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4edf2bfe-ec0e-4469-8401-8f42470ade3b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4edf2bfe-ec0e-4469-8401-8f42470ade3b" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:55:56.958766 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.958740 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4edf2bfe-ec0e-4469-8401-8f42470ade3b" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:55:56.958870 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.958787 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4edf2bfe-ec0e-4469-8401-8f42470ade3b" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:55:56.959019 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.959001 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4edf2bfe-ec0e-4469-8401-8f42470ade3b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4edf2bfe-ec0e-4469-8401-8f42470ade3b" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:55:56.959727 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.959699 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-kube-api-access-vk6lv" (OuterVolumeSpecName: "kube-api-access-vk6lv") pod "4edf2bfe-ec0e-4469-8401-8f42470ade3b" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b"). InnerVolumeSpecName "kube-api-access-vk6lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:55:56.966583 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.966478 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4edf2bfe-ec0e-4469-8401-8f42470ade3b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4edf2bfe-ec0e-4469-8401-8f42470ade3b" (UID: "4edf2bfe-ec0e-4469-8401-8f42470ade3b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:55:56.999067 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.999028 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-2xv2r" event={"ID":"4aad8b73-7fd5-4f1d-89f8-ec9bdb1afb4d","Type":"ContainerStarted","Data":"f262b13e406a4ef61dcd626dadc336af076c9730958b4d2c82f4ea072563857a"} Jul 10 16:55:56.999249 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:56.999138 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-2xv2r" Jul 10 16:55:57.000333 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.000310 2559 generic.go:356] "Generic (PLEG): container finished" podID="4edf2bfe-ec0e-4469-8401-8f42470ade3b" containerID="c37fa068ac1faae78ae8a1f6b4eacf44bc7d5be74c75c30588b8421dd29374fc" exitCode=0 Jul 10 16:55:57.000470 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.000342 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" event={"ID":"4edf2bfe-ec0e-4469-8401-8f42470ade3b","Type":"ContainerDied","Data":"c37fa068ac1faae78ae8a1f6b4eacf44bc7d5be74c75c30588b8421dd29374fc"} Jul 10 16:55:57.000470 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.000393 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" event={"ID":"4edf2bfe-ec0e-4469-8401-8f42470ade3b","Type":"ContainerDied","Data":"cbcfb9936e226562911b2cd11f7a35436081af95829326c60404c8ca34cc2783"} Jul 10 16:55:57.000470 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.000416 2559 scope.go:117] "RemoveContainer" containerID="c37fa068ac1faae78ae8a1f6b4eacf44bc7d5be74c75c30588b8421dd29374fc" Jul 10 16:55:57.000470 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.000352 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd76bfb47-ktthd" Jul 10 16:55:57.000680 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.000638 2559 patch_prober.go:28] interesting pod/downloads-68c9cbb497-2xv2r container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused" start-of-body= Jul 10 16:55:57.000795 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.000703 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-68c9cbb497-2xv2r" podUID="4aad8b73-7fd5-4f1d-89f8-ec9bdb1afb4d" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused" Jul 10 16:55:57.008950 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.008934 2559 scope.go:117] "RemoveContainer" containerID="c37fa068ac1faae78ae8a1f6b4eacf44bc7d5be74c75c30588b8421dd29374fc" Jul 10 16:55:57.009216 ip-10-0-141-228 kubenswrapper[2559]: E0710 16:55:57.009188 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37fa068ac1faae78ae8a1f6b4eacf44bc7d5be74c75c30588b8421dd29374fc\": container with ID starting with c37fa068ac1faae78ae8a1f6b4eacf44bc7d5be74c75c30588b8421dd29374fc not found: ID does not exist" containerID="c37fa068ac1faae78ae8a1f6b4eacf44bc7d5be74c75c30588b8421dd29374fc" Jul 10 16:55:57.009311 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.009223 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37fa068ac1faae78ae8a1f6b4eacf44bc7d5be74c75c30588b8421dd29374fc"} err="failed to get container status \"c37fa068ac1faae78ae8a1f6b4eacf44bc7d5be74c75c30588b8421dd29374fc\": rpc error: code = NotFound desc = could not find container \"c37fa068ac1faae78ae8a1f6b4eacf44bc7d5be74c75c30588b8421dd29374fc\": container with ID starting with c37fa068ac1faae78ae8a1f6b4eacf44bc7d5be74c75c30588b8421dd29374fc not found: ID does not exist" Jul 10 16:55:57.017621 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.017570 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-2xv2r" podStartSLOduration=1.108064915 podStartE2EDuration="19.017553098s" totalImagesPullingTime="17.909488183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:55:57.016473844 +0000 UTC m=+97.142182255" watchObservedRunningTime="2026-07-10 16:55:57.017553098 +0000 UTC m=+97.143261509" Jul 10 16:55:57.033834 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.033800 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cd76bfb47-ktthd"] Jul 10 16:55:57.039381 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.039353 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cd76bfb47-ktthd"] Jul 10 16:55:57.057350 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.057279 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4edf2bfe-ec0e-4469-8401-8f42470ade3b-image-registry-private-configuration\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:55:57.057350 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.057308 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4edf2bfe-ec0e-4469-8401-8f42470ade3b-installation-pull-secrets\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:55:57.057350 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.057320 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk6lv\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-kube-api-access-vk6lv\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:55:57.057350 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.057329 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-tls\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:55:57.057350 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.057338 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4edf2bfe-ec0e-4469-8401-8f42470ade3b-registry-certificates\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:55:57.057350 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.057346 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4edf2bfe-ec0e-4469-8401-8f42470ade3b-bound-sa-token\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:55:57.057350 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.057354 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4edf2bfe-ec0e-4469-8401-8f42470ade3b-ca-trust-extracted\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:55:57.059932 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.059906 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:57.160550 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.160480 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:57.263059 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:57.263020 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:55:58.024270 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:58.024239 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-2xv2r" Jul 10 16:55:58.399736 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:55:58.399704 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4edf2bfe-ec0e-4469-8401-8f42470ade3b" path="/var/lib/kubelet/pods/4edf2bfe-ec0e-4469-8401-8f42470ade3b/volumes" Jul 10 16:56:00.242584 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:00.242539 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:00.411555 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:00.411511 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:01.198681 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:01.198646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:03.211786 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:03.211750 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:03.303735 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:03.303696 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:03.403756 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:03.403721 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:03.489096 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:03.489010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:03.791748 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:03.791646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:03.877100 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:03.877060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:04.799751 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:04.799711 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:04.832896 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:04.832867 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vr2l7" Jul 10 16:56:04.922990 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:04.922954 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:19.543115 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:19.543079 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:19.646990 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:19.646962 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:31.199073 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:31.199036 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:31.261980 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:31.261945 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:56:32.950853 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:56:32.950809 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:57:00.492684 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:00.492640 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:57:01.025543 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:01.025511 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:57:03.485777 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.485743 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 16:57:03.486146 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.486030 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4edf2bfe-ec0e-4469-8401-8f42470ade3b" containerName="registry" Jul 10 16:57:03.486146 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.486040 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4edf2bfe-ec0e-4469-8401-8f42470ade3b" containerName="registry" Jul 10 16:57:03.486146 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.486097 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4edf2bfe-ec0e-4469-8401-8f42470ade3b" containerName="registry" Jul 10 16:57:03.489538 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.489518 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.492349 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.492321 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 16:57:03.492508 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.492360 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bw2t6\"" Jul 10 16:57:03.492508 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.492369 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 16:57:03.492508 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.492400 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 16:57:03.492508 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.492420 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 16:57:03.492508 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.492444 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 16:57:03.492786 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.492770 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 16:57:03.492842 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.492773 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 16:57:03.492911 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.492898 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 16:57:03.514367 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.514334 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 16:57:03.524437 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.524408 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a09795d-9e14-4d7d-a9a9-60b272157376-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.524576 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.524458 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.524576 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.524478 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a09795d-9e14-4d7d-a9a9-60b272157376-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.524655 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.524576 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.524655 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.524598 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.524713 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.524654 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a09795d-9e14-4d7d-a9a9-60b272157376-config-out\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.524713 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.524688 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.524713 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.524706 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a09795d-9e14-4d7d-a9a9-60b272157376-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.524805 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.524730 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.524837 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.524806 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.529358 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.524843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-web-config\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.529358 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.525413 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxqk7\" (UniqueName: \"kubernetes.io/projected/4a09795d-9e14-4d7d-a9a9-60b272157376-kube-api-access-vxqk7\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.529358 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.525463 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a09795d-9e14-4d7d-a9a9-60b272157376-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.541084 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.541048 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 16:57:03.626865 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.626826 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a09795d-9e14-4d7d-a9a9-60b272157376-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.626865 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.626873 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.627125 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.626895 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a09795d-9e14-4d7d-a9a9-60b272157376-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.627125 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.627036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.627125 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.627074 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.627125 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.627117 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a09795d-9e14-4d7d-a9a9-60b272157376-config-out\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.627319 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.627136 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.627319 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.627152 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a09795d-9e14-4d7d-a9a9-60b272157376-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.627319 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.627183 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.627319 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.627237 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.627319 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.627284 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-web-config\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.627319 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.627306 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxqk7\" (UniqueName: \"kubernetes.io/projected/4a09795d-9e14-4d7d-a9a9-60b272157376-kube-api-access-vxqk7\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.627662 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.627330 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a09795d-9e14-4d7d-a9a9-60b272157376-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.627887 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.627744 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a09795d-9e14-4d7d-a9a9-60b272157376-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.627996 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.627970 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a09795d-9e14-4d7d-a9a9-60b272157376-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.629005 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.628979 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a09795d-9e14-4d7d-a9a9-60b272157376-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.630784 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.630378 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.630784 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.630471 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a09795d-9e14-4d7d-a9a9-60b272157376-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.630784 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.630640 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a09795d-9e14-4d7d-a9a9-60b272157376-config-out\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.630784 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.630737 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.630784 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.630768 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.631166 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.631142 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.631240 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.631219 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.631649 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.631630 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-web-config\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.632573 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.632552 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a09795d-9e14-4d7d-a9a9-60b272157376-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.639150 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.639125 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxqk7\" (UniqueName: \"kubernetes.io/projected/4a09795d-9e14-4d7d-a9a9-60b272157376-kube-api-access-vxqk7\") pod \"alertmanager-main-0\" (UID: \"4a09795d-9e14-4d7d-a9a9-60b272157376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.799066 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.798980 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:57:03.941993 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:03.941966 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 16:57:03.944382 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:57:03.944351 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a09795d_9e14_4d7d_a9a9_60b272157376.slice/crio-63029f5ec82ee0fc8e7869518ae00812e8e9db26be733d578f2cb6a337ca8cfe WatchSource:0}: Error finding container 63029f5ec82ee0fc8e7869518ae00812e8e9db26be733d578f2cb6a337ca8cfe: Status 404 returned error can't find the container with id 63029f5ec82ee0fc8e7869518ae00812e8e9db26be733d578f2cb6a337ca8cfe Jul 10 16:57:04.258411 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:04.258374 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a09795d-9e14-4d7d-a9a9-60b272157376","Type":"ContainerStarted","Data":"63029f5ec82ee0fc8e7869518ae00812e8e9db26be733d578f2cb6a337ca8cfe"} Jul 10 16:57:05.264122 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:05.264026 2559 generic.go:356] "Generic (PLEG): container finished" podID="4a09795d-9e14-4d7d-a9a9-60b272157376" containerID="85fa500d13fa67f9f0b0471c01d010189672afc385e362e418f44b3672d9333e" exitCode=0 Jul 10 16:57:05.264122 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:05.264078 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a09795d-9e14-4d7d-a9a9-60b272157376","Type":"ContainerDied","Data":"85fa500d13fa67f9f0b0471c01d010189672afc385e362e418f44b3672d9333e"} Jul 10 16:57:07.276443 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:07.276398 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a09795d-9e14-4d7d-a9a9-60b272157376","Type":"ContainerStarted","Data":"ca46d7961c414876c0dd3bda023cf1edbb84cebf8b63ed38f59bb9cc5f259700"} Jul 10 16:57:07.276794 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:07.276514 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a09795d-9e14-4d7d-a9a9-60b272157376","Type":"ContainerStarted","Data":"06d90c81d5f3ead3da736bc38986525c0d6ac203a5fd94033ebd0253aa0aeeff"} Jul 10 16:57:07.276794 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:07.276543 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a09795d-9e14-4d7d-a9a9-60b272157376","Type":"ContainerStarted","Data":"6570e3b7abf06b200282d9d75ca93e0ec5f06d5d8bb34a3ceb5d4fa32ed289fb"} Jul 10 16:57:07.276794 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:07.276567 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a09795d-9e14-4d7d-a9a9-60b272157376","Type":"ContainerStarted","Data":"0a5043b8c51ce5aee6ec20279fa7aa2289cf07b7910ddfa760708ee0560e7789"} Jul 10 16:57:07.276794 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:07.276598 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a09795d-9e14-4d7d-a9a9-60b272157376","Type":"ContainerStarted","Data":"e319947a409c1a7454849932a70246a07bddc22605b6da685b6ec13c24205de3"} Jul 10 16:57:08.284064 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:08.284028 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a09795d-9e14-4d7d-a9a9-60b272157376","Type":"ContainerStarted","Data":"8aaaf90666da51e5ff81bad625df042898cac9d660444c694d62d92c4b93f899"} Jul 10 16:57:08.325122 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:08.325063 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.613890715 podStartE2EDuration="5.325046483s" totalImagesPullingTime="2.694824517s" totalInitContainerRuntime="16.331251ms" isStatefulPod=false podCreationTimestamp="2026-07-10 16:57:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:57:08.323861505 +0000 UTC m=+168.449569916" watchObservedRunningTime="2026-07-10 16:57:08.325046483 +0000 UTC m=+168.450754895" Jul 10 16:57:10.359795 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:10.359760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:57:10.415305 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:10.415276 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:57:21.647891 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:21.647852 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:57:21.721417 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:21.721381 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:57:59.221929 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:59.221890 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:57:59.275712 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:57:59.275679 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:15.750972 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:15.750934 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:15.802705 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:15.802675 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:16.224437 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:16.224403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:16.288747 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:16.288712 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:16.367951 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:16.367908 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:16.426976 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:16.426934 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:16.967804 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:16.967771 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:17.064822 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:17.064786 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:20.422731 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:20.422697 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:26.833639 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:26.833600 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:26.891711 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:26.891680 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:27.727169 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:27.727132 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:27.810131 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:27.810096 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:34.573334 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:34.573297 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:34.674553 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:34.674520 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:58:42.832410 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:58:42.832338 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:09.950327 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:09.950295 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:10.049941 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.049900 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:10.751879 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.751841 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87"] Jul 10 16:59:10.754463 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.754437 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" Jul 10 16:59:10.756996 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.756976 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8z2vg\"" Jul 10 16:59:10.757254 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.757237 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 16:59:10.757800 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.757784 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 16:59:10.766979 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.766951 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87"] Jul 10 16:59:10.875434 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.875396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsb8c\" (UniqueName: \"kubernetes.io/projected/98cfd6a1-86b8-4316-97f4-152acae04824-kube-api-access-zsb8c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87\" (UID: \"98cfd6a1-86b8-4316-97f4-152acae04824\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" Jul 10 16:59:10.875679 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.875551 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98cfd6a1-86b8-4316-97f4-152acae04824-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87\" (UID: \"98cfd6a1-86b8-4316-97f4-152acae04824\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" Jul 10 16:59:10.875679 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.875594 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98cfd6a1-86b8-4316-97f4-152acae04824-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87\" (UID: \"98cfd6a1-86b8-4316-97f4-152acae04824\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" Jul 10 16:59:10.977007 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.976969 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98cfd6a1-86b8-4316-97f4-152acae04824-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87\" (UID: \"98cfd6a1-86b8-4316-97f4-152acae04824\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" Jul 10 16:59:10.977007 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.977009 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98cfd6a1-86b8-4316-97f4-152acae04824-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87\" (UID: \"98cfd6a1-86b8-4316-97f4-152acae04824\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" Jul 10 16:59:10.977565 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.977036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsb8c\" (UniqueName: \"kubernetes.io/projected/98cfd6a1-86b8-4316-97f4-152acae04824-kube-api-access-zsb8c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87\" (UID: \"98cfd6a1-86b8-4316-97f4-152acae04824\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" Jul 10 16:59:10.977565 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.977370 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98cfd6a1-86b8-4316-97f4-152acae04824-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87\" (UID: \"98cfd6a1-86b8-4316-97f4-152acae04824\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" Jul 10 16:59:10.977565 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.977442 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98cfd6a1-86b8-4316-97f4-152acae04824-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87\" (UID: \"98cfd6a1-86b8-4316-97f4-152acae04824\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" Jul 10 16:59:10.985427 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:10.985396 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsb8c\" (UniqueName: \"kubernetes.io/projected/98cfd6a1-86b8-4316-97f4-152acae04824-kube-api-access-zsb8c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87\" (UID: \"98cfd6a1-86b8-4316-97f4-152acae04824\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" Jul 10 16:59:11.064540 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:11.064416 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" Jul 10 16:59:11.200631 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:11.200596 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87"] Jul 10 16:59:11.203833 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:59:11.203804 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cfd6a1_86b8_4316_97f4_152acae04824.slice/crio-81ca2b4087ddabd580fdcbc02e223d4d95a2fad274ac35f4d204b4c46fa706b0 WatchSource:0}: Error finding container 81ca2b4087ddabd580fdcbc02e223d4d95a2fad274ac35f4d204b4c46fa706b0: Status 404 returned error can't find the container with id 81ca2b4087ddabd580fdcbc02e223d4d95a2fad274ac35f4d204b4c46fa706b0 Jul 10 16:59:11.760096 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:11.760058 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" event={"ID":"98cfd6a1-86b8-4316-97f4-152acae04824","Type":"ContainerStarted","Data":"81ca2b4087ddabd580fdcbc02e223d4d95a2fad274ac35f4d204b4c46fa706b0"} Jul 10 16:59:16.783966 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:16.783881 2559 generic.go:356] "Generic (PLEG): container finished" podID="98cfd6a1-86b8-4316-97f4-152acae04824" containerID="c85f364db6529897c73c3138c8ab93e21f99921f921cde091fa683ce78c497c7" exitCode=0 Jul 10 16:59:16.784322 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:16.783972 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" event={"ID":"98cfd6a1-86b8-4316-97f4-152acae04824","Type":"ContainerDied","Data":"c85f364db6529897c73c3138c8ab93e21f99921f921cde091fa683ce78c497c7"} Jul 10 16:59:19.797663 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:19.797624 2559 generic.go:356] "Generic (PLEG): container finished" podID="98cfd6a1-86b8-4316-97f4-152acae04824" containerID="6b02d5288d6716d574d7dd1311badcff5051686bf7917ac5ca5f2b72d729d18e" exitCode=0 Jul 10 16:59:19.798141 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:19.797685 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" event={"ID":"98cfd6a1-86b8-4316-97f4-152acae04824","Type":"ContainerDied","Data":"6b02d5288d6716d574d7dd1311badcff5051686bf7917ac5ca5f2b72d729d18e"} Jul 10 16:59:20.279597 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:20.279568 2559 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 16:59:22.244287 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:22.244249 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:23.432986 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:23.432949 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:28.836957 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:28.836925 2559 generic.go:356] "Generic (PLEG): container finished" podID="98cfd6a1-86b8-4316-97f4-152acae04824" containerID="03e87a13d282e7ccfaf9a3c6fffab66660f7d40aaac67c3bfe392595f50d7b6e" exitCode=0 Jul 10 16:59:28.837327 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:28.836984 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" event={"ID":"98cfd6a1-86b8-4316-97f4-152acae04824","Type":"ContainerDied","Data":"03e87a13d282e7ccfaf9a3c6fffab66660f7d40aaac67c3bfe392595f50d7b6e"} Jul 10 16:59:29.967592 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:29.967567 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" Jul 10 16:59:30.151822 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:30.151731 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98cfd6a1-86b8-4316-97f4-152acae04824-util\") pod \"98cfd6a1-86b8-4316-97f4-152acae04824\" (UID: \"98cfd6a1-86b8-4316-97f4-152acae04824\") " Jul 10 16:59:30.151822 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:30.151782 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98cfd6a1-86b8-4316-97f4-152acae04824-bundle\") pod \"98cfd6a1-86b8-4316-97f4-152acae04824\" (UID: \"98cfd6a1-86b8-4316-97f4-152acae04824\") " Jul 10 16:59:30.151822 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:30.151825 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsb8c\" (UniqueName: \"kubernetes.io/projected/98cfd6a1-86b8-4316-97f4-152acae04824-kube-api-access-zsb8c\") pod \"98cfd6a1-86b8-4316-97f4-152acae04824\" (UID: \"98cfd6a1-86b8-4316-97f4-152acae04824\") " Jul 10 16:59:30.152519 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:30.152472 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cfd6a1-86b8-4316-97f4-152acae04824-bundle" (OuterVolumeSpecName: "bundle") pod "98cfd6a1-86b8-4316-97f4-152acae04824" (UID: "98cfd6a1-86b8-4316-97f4-152acae04824"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:59:30.154174 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:30.154151 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cfd6a1-86b8-4316-97f4-152acae04824-kube-api-access-zsb8c" (OuterVolumeSpecName: "kube-api-access-zsb8c") pod "98cfd6a1-86b8-4316-97f4-152acae04824" (UID: "98cfd6a1-86b8-4316-97f4-152acae04824"). InnerVolumeSpecName "kube-api-access-zsb8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:59:30.157075 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:30.157048 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cfd6a1-86b8-4316-97f4-152acae04824-util" (OuterVolumeSpecName: "util") pod "98cfd6a1-86b8-4316-97f4-152acae04824" (UID: "98cfd6a1-86b8-4316-97f4-152acae04824"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:59:30.252849 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:30.252798 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsb8c\" (UniqueName: \"kubernetes.io/projected/98cfd6a1-86b8-4316-97f4-152acae04824-kube-api-access-zsb8c\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:59:30.252849 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:30.252842 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98cfd6a1-86b8-4316-97f4-152acae04824-util\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:59:30.252849 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:30.252851 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98cfd6a1-86b8-4316-97f4-152acae04824-bundle\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:59:30.847031 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:30.846994 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" event={"ID":"98cfd6a1-86b8-4316-97f4-152acae04824","Type":"ContainerDied","Data":"81ca2b4087ddabd580fdcbc02e223d4d95a2fad274ac35f4d204b4c46fa706b0"} Jul 10 16:59:30.847191 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:30.847048 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81ca2b4087ddabd580fdcbc02e223d4d95a2fad274ac35f4d204b4c46fa706b0" Jul 10 16:59:30.847191 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:30.847012 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glp87" Jul 10 16:59:36.834228 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:36.834190 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:37.667668 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:37.667635 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:40.471111 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:40.471079 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:40.540549 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:40.540485 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:42.288739 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.288702 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:42.394530 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.394480 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:42.562119 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.562032 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:42.704369 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.704339 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:42.721225 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.721188 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hzhsf"] Jul 10 16:59:42.721539 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.721526 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98cfd6a1-86b8-4316-97f4-152acae04824" containerName="extract" Jul 10 16:59:42.721588 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.721542 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="98cfd6a1-86b8-4316-97f4-152acae04824" containerName="extract" Jul 10 16:59:42.721588 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.721552 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98cfd6a1-86b8-4316-97f4-152acae04824" containerName="util" Jul 10 16:59:42.721588 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.721558 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="98cfd6a1-86b8-4316-97f4-152acae04824" containerName="util" Jul 10 16:59:42.721588 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.721570 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98cfd6a1-86b8-4316-97f4-152acae04824" containerName="pull" Jul 10 16:59:42.721588 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.721576 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="98cfd6a1-86b8-4316-97f4-152acae04824" containerName="pull" Jul 10 16:59:42.721733 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.721623 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="98cfd6a1-86b8-4316-97f4-152acae04824" containerName="extract" Jul 10 16:59:42.723729 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.723712 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hzhsf" Jul 10 16:59:42.726332 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.726311 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 16:59:42.726470 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.726337 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-xp9fs\"" Jul 10 16:59:42.727421 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.727407 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 16:59:42.744594 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.744557 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6bab174e-b40d-447c-a521-47639694854e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hzhsf\" (UID: \"6bab174e-b40d-447c-a521-47639694854e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hzhsf" Jul 10 16:59:42.744795 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.744773 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhmj2\" (UniqueName: \"kubernetes.io/projected/6bab174e-b40d-447c-a521-47639694854e-kube-api-access-lhmj2\") pod \"cert-manager-webhook-6486ffd997-hzhsf\" (UID: \"6bab174e-b40d-447c-a521-47639694854e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hzhsf" Jul 10 16:59:42.746087 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.746061 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hzhsf"] Jul 10 16:59:42.845763 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.845727 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6bab174e-b40d-447c-a521-47639694854e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hzhsf\" (UID: \"6bab174e-b40d-447c-a521-47639694854e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hzhsf" Jul 10 16:59:42.845915 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.845781 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhmj2\" (UniqueName: \"kubernetes.io/projected/6bab174e-b40d-447c-a521-47639694854e-kube-api-access-lhmj2\") pod \"cert-manager-webhook-6486ffd997-hzhsf\" (UID: \"6bab174e-b40d-447c-a521-47639694854e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hzhsf" Jul 10 16:59:42.862425 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.862387 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6bab174e-b40d-447c-a521-47639694854e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hzhsf\" (UID: \"6bab174e-b40d-447c-a521-47639694854e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hzhsf" Jul 10 16:59:42.862575 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:42.862512 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhmj2\" (UniqueName: \"kubernetes.io/projected/6bab174e-b40d-447c-a521-47639694854e-kube-api-access-lhmj2\") pod \"cert-manager-webhook-6486ffd997-hzhsf\" (UID: \"6bab174e-b40d-447c-a521-47639694854e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hzhsf" Jul 10 16:59:43.032658 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.032615 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hzhsf" Jul 10 16:59:43.132697 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.132665 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd"] Jul 10 16:59:43.136514 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.136266 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" Jul 10 16:59:43.139078 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.139039 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8z2vg\"" Jul 10 16:59:43.139235 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.139116 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 16:59:43.139235 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.139038 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 16:59:43.146945 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.146909 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd"] Jul 10 16:59:43.147972 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.147950 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/354d8cb6-55fc-4196-a9b8-589b752fd469-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd\" (UID: \"354d8cb6-55fc-4196-a9b8-589b752fd469\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" Jul 10 16:59:43.148085 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.148003 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/354d8cb6-55fc-4196-a9b8-589b752fd469-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd\" (UID: \"354d8cb6-55fc-4196-a9b8-589b752fd469\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" Jul 10 16:59:43.148085 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.148029 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cntdw\" (UniqueName: \"kubernetes.io/projected/354d8cb6-55fc-4196-a9b8-589b752fd469-kube-api-access-cntdw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd\" (UID: \"354d8cb6-55fc-4196-a9b8-589b752fd469\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" Jul 10 16:59:43.180954 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.180927 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hzhsf"] Jul 10 16:59:43.183775 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:59:43.183745 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bab174e_b40d_447c_a521_47639694854e.slice/crio-932a3562f7335debd4929c0f4a606df47da845b4d064761199758a37255a7dd6 WatchSource:0}: Error finding container 932a3562f7335debd4929c0f4a606df47da845b4d064761199758a37255a7dd6: Status 404 returned error can't find the container with id 932a3562f7335debd4929c0f4a606df47da845b4d064761199758a37255a7dd6 Jul 10 16:59:43.185839 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.185818 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:59:43.248647 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.248599 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/354d8cb6-55fc-4196-a9b8-589b752fd469-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd\" (UID: \"354d8cb6-55fc-4196-a9b8-589b752fd469\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" Jul 10 16:59:43.248647 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.248651 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cntdw\" (UniqueName: \"kubernetes.io/projected/354d8cb6-55fc-4196-a9b8-589b752fd469-kube-api-access-cntdw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd\" (UID: \"354d8cb6-55fc-4196-a9b8-589b752fd469\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" Jul 10 16:59:43.248908 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.248715 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/354d8cb6-55fc-4196-a9b8-589b752fd469-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd\" (UID: \"354d8cb6-55fc-4196-a9b8-589b752fd469\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" Jul 10 16:59:43.249055 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.249031 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/354d8cb6-55fc-4196-a9b8-589b752fd469-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd\" (UID: \"354d8cb6-55fc-4196-a9b8-589b752fd469\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" Jul 10 16:59:43.249117 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.249039 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/354d8cb6-55fc-4196-a9b8-589b752fd469-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd\" (UID: \"354d8cb6-55fc-4196-a9b8-589b752fd469\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" Jul 10 16:59:43.258109 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.258084 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cntdw\" (UniqueName: \"kubernetes.io/projected/354d8cb6-55fc-4196-a9b8-589b752fd469-kube-api-access-cntdw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd\" (UID: \"354d8cb6-55fc-4196-a9b8-589b752fd469\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" Jul 10 16:59:43.447824 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.447737 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" Jul 10 16:59:43.588018 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.587990 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd"] Jul 10 16:59:43.589940 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:59:43.589912 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354d8cb6_55fc_4196_a9b8_589b752fd469.slice/crio-4848e53b726619ec12fb4b961ce1fbaf42194a8db0b337861fabeb5c38eac018 WatchSource:0}: Error finding container 4848e53b726619ec12fb4b961ce1fbaf42194a8db0b337861fabeb5c38eac018: Status 404 returned error can't find the container with id 4848e53b726619ec12fb4b961ce1fbaf42194a8db0b337861fabeb5c38eac018 Jul 10 16:59:43.901375 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.901340 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hzhsf" event={"ID":"6bab174e-b40d-447c-a521-47639694854e","Type":"ContainerStarted","Data":"932a3562f7335debd4929c0f4a606df47da845b4d064761199758a37255a7dd6"} Jul 10 16:59:43.902746 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.902723 2559 generic.go:356] "Generic (PLEG): container finished" podID="354d8cb6-55fc-4196-a9b8-589b752fd469" containerID="29bc1e8be18fc554c76c2e113a9aeed0d3deafe3bd067d8a5c6d76c8bd1c9caa" exitCode=0 Jul 10 16:59:43.902864 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.902753 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" event={"ID":"354d8cb6-55fc-4196-a9b8-589b752fd469","Type":"ContainerDied","Data":"29bc1e8be18fc554c76c2e113a9aeed0d3deafe3bd067d8a5c6d76c8bd1c9caa"} Jul 10 16:59:43.902864 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:43.902789 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" event={"ID":"354d8cb6-55fc-4196-a9b8-589b752fd469","Type":"ContainerStarted","Data":"4848e53b726619ec12fb4b961ce1fbaf42194a8db0b337861fabeb5c38eac018"} Jul 10 16:59:44.672141 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:44.672037 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:45.151841 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:45.151807 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:46.925904 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:46.925616 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hzhsf" event={"ID":"6bab174e-b40d-447c-a521-47639694854e","Type":"ContainerStarted","Data":"130209b31a89fc531445824224709cd7baec0c7dbefc354cb02c50b2becbe1a7"} Jul 10 16:59:46.925904 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:46.925680 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hzhsf" Jul 10 16:59:46.927834 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:46.927807 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" event={"ID":"354d8cb6-55fc-4196-a9b8-589b752fd469","Type":"ContainerStarted","Data":"da257681e6218dbddd7034c439647406114b7c3baa4b1148c6e4df78cb2cb4e0"} Jul 10 16:59:46.946684 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:46.946638 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-hzhsf" podStartSLOduration=1.334226101 podStartE2EDuration="4.946621721s" totalImagesPullingTime="3.61239562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:59:46.944619111 +0000 UTC m=+327.070327520" watchObservedRunningTime="2026-07-10 16:59:46.946621721 +0000 UTC m=+327.072330133" Jul 10 16:59:47.384631 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:47.384601 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:47.477856 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:47.477816 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:47.934208 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:47.934169 2559 generic.go:356] "Generic (PLEG): container finished" podID="354d8cb6-55fc-4196-a9b8-589b752fd469" containerID="da257681e6218dbddd7034c439647406114b7c3baa4b1148c6e4df78cb2cb4e0" exitCode=0 Jul 10 16:59:47.934675 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:47.934250 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" event={"ID":"354d8cb6-55fc-4196-a9b8-589b752fd469","Type":"ContainerDied","Data":"da257681e6218dbddd7034c439647406114b7c3baa4b1148c6e4df78cb2cb4e0"} Jul 10 16:59:48.940124 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:48.940091 2559 generic.go:356] "Generic (PLEG): container finished" podID="354d8cb6-55fc-4196-a9b8-589b752fd469" containerID="687c6cbfa13a87174b9201f3a299590aacbcfb1ee894dbe5e04b2437f96b0eb8" exitCode=0 Jul 10 16:59:48.940522 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:48.940174 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" event={"ID":"354d8cb6-55fc-4196-a9b8-589b752fd469","Type":"ContainerDied","Data":"687c6cbfa13a87174b9201f3a299590aacbcfb1ee894dbe5e04b2437f96b0eb8"} Jul 10 16:59:50.075310 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:50.075282 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" Jul 10 16:59:50.100666 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:50.100635 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/354d8cb6-55fc-4196-a9b8-589b752fd469-bundle\") pod \"354d8cb6-55fc-4196-a9b8-589b752fd469\" (UID: \"354d8cb6-55fc-4196-a9b8-589b752fd469\") " Jul 10 16:59:50.100841 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:50.100769 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/354d8cb6-55fc-4196-a9b8-589b752fd469-util\") pod \"354d8cb6-55fc-4196-a9b8-589b752fd469\" (UID: \"354d8cb6-55fc-4196-a9b8-589b752fd469\") " Jul 10 16:59:50.100841 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:50.100789 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cntdw\" (UniqueName: \"kubernetes.io/projected/354d8cb6-55fc-4196-a9b8-589b752fd469-kube-api-access-cntdw\") pod \"354d8cb6-55fc-4196-a9b8-589b752fd469\" (UID: \"354d8cb6-55fc-4196-a9b8-589b752fd469\") " Jul 10 16:59:50.101295 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:50.101113 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354d8cb6-55fc-4196-a9b8-589b752fd469-bundle" (OuterVolumeSpecName: "bundle") pod "354d8cb6-55fc-4196-a9b8-589b752fd469" (UID: "354d8cb6-55fc-4196-a9b8-589b752fd469"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:59:50.103159 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:50.103124 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354d8cb6-55fc-4196-a9b8-589b752fd469-kube-api-access-cntdw" (OuterVolumeSpecName: "kube-api-access-cntdw") pod "354d8cb6-55fc-4196-a9b8-589b752fd469" (UID: "354d8cb6-55fc-4196-a9b8-589b752fd469"). InnerVolumeSpecName "kube-api-access-cntdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:59:50.107936 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:50.107901 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354d8cb6-55fc-4196-a9b8-589b752fd469-util" (OuterVolumeSpecName: "util") pod "354d8cb6-55fc-4196-a9b8-589b752fd469" (UID: "354d8cb6-55fc-4196-a9b8-589b752fd469"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:59:50.201923 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:50.201830 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/354d8cb6-55fc-4196-a9b8-589b752fd469-util\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:59:50.201923 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:50.201865 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cntdw\" (UniqueName: \"kubernetes.io/projected/354d8cb6-55fc-4196-a9b8-589b752fd469-kube-api-access-cntdw\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:59:50.201923 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:50.201877 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/354d8cb6-55fc-4196-a9b8-589b752fd469-bundle\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 16:59:50.950820 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:50.950781 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" event={"ID":"354d8cb6-55fc-4196-a9b8-589b752fd469","Type":"ContainerDied","Data":"4848e53b726619ec12fb4b961ce1fbaf42194a8db0b337861fabeb5c38eac018"} Jul 10 16:59:50.950990 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:50.950834 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4848e53b726619ec12fb4b961ce1fbaf42194a8db0b337861fabeb5c38eac018" Jul 10 16:59:50.950990 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:50.950793 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjd" Jul 10 16:59:52.882370 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:52.882334 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:52.936935 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:52.936907 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hzhsf" Jul 10 16:59:53.526461 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.526421 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7hv82"] Jul 10 16:59:53.526844 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.526826 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354d8cb6-55fc-4196-a9b8-589b752fd469" containerName="pull" Jul 10 16:59:53.526928 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.526847 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="354d8cb6-55fc-4196-a9b8-589b752fd469" containerName="pull" Jul 10 16:59:53.526928 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.526878 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354d8cb6-55fc-4196-a9b8-589b752fd469" containerName="util" Jul 10 16:59:53.526928 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.526886 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="354d8cb6-55fc-4196-a9b8-589b752fd469" containerName="util" Jul 10 16:59:53.526928 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.526897 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354d8cb6-55fc-4196-a9b8-589b752fd469" containerName="extract" Jul 10 16:59:53.526928 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.526906 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="354d8cb6-55fc-4196-a9b8-589b752fd469" containerName="extract" Jul 10 16:59:53.527148 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.526988 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="354d8cb6-55fc-4196-a9b8-589b752fd469" containerName="extract" Jul 10 16:59:53.532479 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.532454 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7hv82" Jul 10 16:59:53.536001 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.535985 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-tws4h\"" Jul 10 16:59:53.553030 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.553002 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7hv82"] Jul 10 16:59:53.629126 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.629080 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4504a8c7-bcbe-4a87-a58d-ada432501edd-bound-sa-token\") pod \"cert-manager-55b4d954d5-7hv82\" (UID: \"4504a8c7-bcbe-4a87-a58d-ada432501edd\") " pod="cert-manager/cert-manager-55b4d954d5-7hv82" Jul 10 16:59:53.629313 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.629138 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkrrr\" (UniqueName: \"kubernetes.io/projected/4504a8c7-bcbe-4a87-a58d-ada432501edd-kube-api-access-lkrrr\") pod \"cert-manager-55b4d954d5-7hv82\" (UID: \"4504a8c7-bcbe-4a87-a58d-ada432501edd\") " pod="cert-manager/cert-manager-55b4d954d5-7hv82" Jul 10 16:59:53.729826 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.729790 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4504a8c7-bcbe-4a87-a58d-ada432501edd-bound-sa-token\") pod \"cert-manager-55b4d954d5-7hv82\" (UID: \"4504a8c7-bcbe-4a87-a58d-ada432501edd\") " pod="cert-manager/cert-manager-55b4d954d5-7hv82" Jul 10 16:59:53.729933 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.729840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkrrr\" (UniqueName: \"kubernetes.io/projected/4504a8c7-bcbe-4a87-a58d-ada432501edd-kube-api-access-lkrrr\") pod \"cert-manager-55b4d954d5-7hv82\" (UID: \"4504a8c7-bcbe-4a87-a58d-ada432501edd\") " pod="cert-manager/cert-manager-55b4d954d5-7hv82" Jul 10 16:59:53.742032 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.741992 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4504a8c7-bcbe-4a87-a58d-ada432501edd-bound-sa-token\") pod \"cert-manager-55b4d954d5-7hv82\" (UID: \"4504a8c7-bcbe-4a87-a58d-ada432501edd\") " pod="cert-manager/cert-manager-55b4d954d5-7hv82" Jul 10 16:59:53.742155 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.742120 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkrrr\" (UniqueName: \"kubernetes.io/projected/4504a8c7-bcbe-4a87-a58d-ada432501edd-kube-api-access-lkrrr\") pod \"cert-manager-55b4d954d5-7hv82\" (UID: \"4504a8c7-bcbe-4a87-a58d-ada432501edd\") " pod="cert-manager/cert-manager-55b4d954d5-7hv82" Jul 10 16:59:53.841764 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.841729 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7hv82" Jul 10 16:59:53.977129 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:53.977105 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7hv82"] Jul 10 16:59:53.979785 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:59:53.979759 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4504a8c7_bcbe_4a87_a58d_ada432501edd.slice/crio-efcd3d0d0689d4342fac65400106b3a1a5a4ef973117d3b8f081bd7c2a073138 WatchSource:0}: Error finding container efcd3d0d0689d4342fac65400106b3a1a5a4ef973117d3b8f081bd7c2a073138: Status 404 returned error can't find the container with id efcd3d0d0689d4342fac65400106b3a1a5a4ef973117d3b8f081bd7c2a073138 Jul 10 16:59:54.932239 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:54.932203 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:54.967403 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:54.967352 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7hv82" event={"ID":"4504a8c7-bcbe-4a87-a58d-ada432501edd","Type":"ContainerStarted","Data":"d35aee40ac6f940b6d90cfcb6cbca63ee6306fc4e5861abebfe883f699e088e6"} Jul 10 16:59:54.967586 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:54.967410 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7hv82" event={"ID":"4504a8c7-bcbe-4a87-a58d-ada432501edd","Type":"ContainerStarted","Data":"efcd3d0d0689d4342fac65400106b3a1a5a4ef973117d3b8f081bd7c2a073138"} Jul 10 16:59:54.987779 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:54.987723 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-7hv82" podStartSLOduration=1.987706868 podStartE2EDuration="1.987706868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:59:54.985256773 +0000 UTC m=+335.110965183" watchObservedRunningTime="2026-07-10 16:59:54.987706868 +0000 UTC m=+335.113415289" Jul 10 16:59:55.032343 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:55.032308 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:56.139865 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.139830 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:56.241804 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.241772 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 16:59:56.306227 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.306172 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7"] Jul 10 16:59:56.310780 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.310748 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7" Jul 10 16:59:56.313539 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.313517 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 16:59:56.314526 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.314486 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-mmfg5\"" Jul 10 16:59:56.314747 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.314733 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:59:56.322978 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.322957 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7"] Jul 10 16:59:56.351515 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.351465 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zsv2\" (UniqueName: \"kubernetes.io/projected/1afe131e-7370-4941-9bcd-377e4ead5488-kube-api-access-6zsv2\") pod \"openshift-lws-operator-fd8ccff4c-fmdg7\" (UID: \"1afe131e-7370-4941-9bcd-377e4ead5488\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7" Jul 10 16:59:56.351767 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.351613 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1afe131e-7370-4941-9bcd-377e4ead5488-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fmdg7\" (UID: \"1afe131e-7370-4941-9bcd-377e4ead5488\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7" Jul 10 16:59:56.452718 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.452614 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1afe131e-7370-4941-9bcd-377e4ead5488-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fmdg7\" (UID: \"1afe131e-7370-4941-9bcd-377e4ead5488\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7" Jul 10 16:59:56.452718 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.452683 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zsv2\" (UniqueName: \"kubernetes.io/projected/1afe131e-7370-4941-9bcd-377e4ead5488-kube-api-access-6zsv2\") pod \"openshift-lws-operator-fd8ccff4c-fmdg7\" (UID: \"1afe131e-7370-4941-9bcd-377e4ead5488\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7" Jul 10 16:59:56.453017 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.452996 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1afe131e-7370-4941-9bcd-377e4ead5488-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fmdg7\" (UID: \"1afe131e-7370-4941-9bcd-377e4ead5488\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7" Jul 10 16:59:56.467754 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.467723 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zsv2\" (UniqueName: \"kubernetes.io/projected/1afe131e-7370-4941-9bcd-377e4ead5488-kube-api-access-6zsv2\") pod \"openshift-lws-operator-fd8ccff4c-fmdg7\" (UID: \"1afe131e-7370-4941-9bcd-377e4ead5488\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7" Jul 10 16:59:56.623341 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.623301 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7" Jul 10 16:59:56.763616 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.763591 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7"] Jul 10 16:59:56.766223 ip-10-0-141-228 kubenswrapper[2559]: W0710 16:59:56.766193 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1afe131e_7370_4941_9bcd_377e4ead5488.slice/crio-a48c6ecab0056ab24412cb43c67106d46faa4f24ce20f772294a9bb00a340ae8 WatchSource:0}: Error finding container a48c6ecab0056ab24412cb43c67106d46faa4f24ce20f772294a9bb00a340ae8: Status 404 returned error can't find the container with id a48c6ecab0056ab24412cb43c67106d46faa4f24ce20f772294a9bb00a340ae8 Jul 10 16:59:56.977327 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:56.977244 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7" event={"ID":"1afe131e-7370-4941-9bcd-377e4ead5488","Type":"ContainerStarted","Data":"a48c6ecab0056ab24412cb43c67106d46faa4f24ce20f772294a9bb00a340ae8"} Jul 10 16:59:59.993947 ip-10-0-141-228 kubenswrapper[2559]: I0710 16:59:59.993909 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7" event={"ID":"1afe131e-7370-4941-9bcd-377e4ead5488","Type":"ContainerStarted","Data":"df3ff5d76b774353569d61665c87b33e035f18e872f23e4377012d7df8b6a3c2"} Jul 10 17:00:00.013007 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:00.012958 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmdg7" podStartSLOduration=1.5717806159999999 podStartE2EDuration="4.012941457s" totalImagesPullingTime="2.441160841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:59:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:00:00.011808282 +0000 UTC m=+340.137516696" watchObservedRunningTime="2026-07-10 17:00:00.012941457 +0000 UTC m=+340.138649867" Jul 10 17:00:00.853792 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:00.853758 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:00.954295 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:00.954258 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:02.731842 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:02.731807 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:02.821069 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:02.821029 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2"] Jul 10 17:00:02.823759 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:02.823736 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" Jul 10 17:00:02.826567 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:02.826545 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 17:00:02.826700 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:02.826565 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 17:00:02.827626 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:02.827611 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8z2vg\"" Jul 10 17:00:02.832594 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:02.832573 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2"] Jul 10 17:00:02.907985 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:02.907941 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgwjp\" (UniqueName: \"kubernetes.io/projected/827af078-68a9-4e0e-92c1-dd7e6e337052-kube-api-access-mgwjp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2\" (UID: \"827af078-68a9-4e0e-92c1-dd7e6e337052\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" Jul 10 17:00:02.907985 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:02.907990 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/827af078-68a9-4e0e-92c1-dd7e6e337052-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2\" (UID: \"827af078-68a9-4e0e-92c1-dd7e6e337052\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" Jul 10 17:00:02.908210 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:02.908015 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/827af078-68a9-4e0e-92c1-dd7e6e337052-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2\" (UID: \"827af078-68a9-4e0e-92c1-dd7e6e337052\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" Jul 10 17:00:03.009317 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:03.009236 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwjp\" (UniqueName: \"kubernetes.io/projected/827af078-68a9-4e0e-92c1-dd7e6e337052-kube-api-access-mgwjp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2\" (UID: \"827af078-68a9-4e0e-92c1-dd7e6e337052\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" Jul 10 17:00:03.009317 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:03.009269 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/827af078-68a9-4e0e-92c1-dd7e6e337052-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2\" (UID: \"827af078-68a9-4e0e-92c1-dd7e6e337052\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" Jul 10 17:00:03.009317 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:03.009295 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/827af078-68a9-4e0e-92c1-dd7e6e337052-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2\" (UID: \"827af078-68a9-4e0e-92c1-dd7e6e337052\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" Jul 10 17:00:03.009649 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:03.009630 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/827af078-68a9-4e0e-92c1-dd7e6e337052-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2\" (UID: \"827af078-68a9-4e0e-92c1-dd7e6e337052\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" Jul 10 17:00:03.009687 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:03.009663 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/827af078-68a9-4e0e-92c1-dd7e6e337052-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2\" (UID: \"827af078-68a9-4e0e-92c1-dd7e6e337052\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" Jul 10 17:00:03.019031 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:03.019003 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwjp\" (UniqueName: \"kubernetes.io/projected/827af078-68a9-4e0e-92c1-dd7e6e337052-kube-api-access-mgwjp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2\" (UID: \"827af078-68a9-4e0e-92c1-dd7e6e337052\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" Jul 10 17:00:03.133276 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:03.133237 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" Jul 10 17:00:03.271890 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:03.271791 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2"] Jul 10 17:00:03.274424 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:00:03.274393 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827af078_68a9_4e0e_92c1_dd7e6e337052.slice/crio-7a82e6fbed030875f370c4cdf32867003c80232060b73c48c82aca534a36d2fa WatchSource:0}: Error finding container 7a82e6fbed030875f370c4cdf32867003c80232060b73c48c82aca534a36d2fa: Status 404 returned error can't find the container with id 7a82e6fbed030875f370c4cdf32867003c80232060b73c48c82aca534a36d2fa Jul 10 17:00:04.012820 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:04.012786 2559 generic.go:356] "Generic (PLEG): container finished" podID="827af078-68a9-4e0e-92c1-dd7e6e337052" containerID="cfb7bccc7bd8df6a47594b9dc7fdfd294a665f4ab05cb5e62a0fa15c259a35eb" exitCode=0 Jul 10 17:00:04.013180 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:04.012874 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" event={"ID":"827af078-68a9-4e0e-92c1-dd7e6e337052","Type":"ContainerDied","Data":"cfb7bccc7bd8df6a47594b9dc7fdfd294a665f4ab05cb5e62a0fa15c259a35eb"} Jul 10 17:00:04.013180 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:04.012949 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" event={"ID":"827af078-68a9-4e0e-92c1-dd7e6e337052","Type":"ContainerStarted","Data":"7a82e6fbed030875f370c4cdf32867003c80232060b73c48c82aca534a36d2fa"} Jul 10 17:00:06.024011 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:06.023975 2559 generic.go:356] "Generic (PLEG): container finished" podID="827af078-68a9-4e0e-92c1-dd7e6e337052" containerID="13e9dade6a653546736e80fae65b5de5ca9ec2647539f13d0533a6e07e924b9b" exitCode=0 Jul 10 17:00:06.024482 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:06.024069 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" event={"ID":"827af078-68a9-4e0e-92c1-dd7e6e337052","Type":"ContainerDied","Data":"13e9dade6a653546736e80fae65b5de5ca9ec2647539f13d0533a6e07e924b9b"} Jul 10 17:00:07.030893 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:07.030861 2559 generic.go:356] "Generic (PLEG): container finished" podID="827af078-68a9-4e0e-92c1-dd7e6e337052" containerID="c4b2f9094a8ac4aca555db5fd0a4ce322713da8125bc80ef5f537f986f30fcc3" exitCode=0 Jul 10 17:00:07.031387 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:07.030950 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" event={"ID":"827af078-68a9-4e0e-92c1-dd7e6e337052","Type":"ContainerDied","Data":"c4b2f9094a8ac4aca555db5fd0a4ce322713da8125bc80ef5f537f986f30fcc3"} Jul 10 17:00:08.171118 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:08.171090 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" Jul 10 17:00:08.254742 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:08.254710 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgwjp\" (UniqueName: \"kubernetes.io/projected/827af078-68a9-4e0e-92c1-dd7e6e337052-kube-api-access-mgwjp\") pod \"827af078-68a9-4e0e-92c1-dd7e6e337052\" (UID: \"827af078-68a9-4e0e-92c1-dd7e6e337052\") " Jul 10 17:00:08.254903 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:08.254760 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/827af078-68a9-4e0e-92c1-dd7e6e337052-bundle\") pod \"827af078-68a9-4e0e-92c1-dd7e6e337052\" (UID: \"827af078-68a9-4e0e-92c1-dd7e6e337052\") " Jul 10 17:00:08.254903 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:08.254820 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/827af078-68a9-4e0e-92c1-dd7e6e337052-util\") pod \"827af078-68a9-4e0e-92c1-dd7e6e337052\" (UID: \"827af078-68a9-4e0e-92c1-dd7e6e337052\") " Jul 10 17:00:08.255552 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:08.255524 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827af078-68a9-4e0e-92c1-dd7e6e337052-bundle" (OuterVolumeSpecName: "bundle") pod "827af078-68a9-4e0e-92c1-dd7e6e337052" (UID: "827af078-68a9-4e0e-92c1-dd7e6e337052"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:00:08.257164 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:08.257143 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827af078-68a9-4e0e-92c1-dd7e6e337052-kube-api-access-mgwjp" (OuterVolumeSpecName: "kube-api-access-mgwjp") pod "827af078-68a9-4e0e-92c1-dd7e6e337052" (UID: "827af078-68a9-4e0e-92c1-dd7e6e337052"). InnerVolumeSpecName "kube-api-access-mgwjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:00:08.260481 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:08.260451 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827af078-68a9-4e0e-92c1-dd7e6e337052-util" (OuterVolumeSpecName: "util") pod "827af078-68a9-4e0e-92c1-dd7e6e337052" (UID: "827af078-68a9-4e0e-92c1-dd7e6e337052"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:00:08.355999 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:08.355964 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/827af078-68a9-4e0e-92c1-dd7e6e337052-util\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:00:08.355999 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:08.355998 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgwjp\" (UniqueName: \"kubernetes.io/projected/827af078-68a9-4e0e-92c1-dd7e6e337052-kube-api-access-mgwjp\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:00:08.356210 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:08.356009 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/827af078-68a9-4e0e-92c1-dd7e6e337052-bundle\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:00:09.043072 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:09.043041 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" Jul 10 17:00:09.043245 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:09.043044 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflskq2" event={"ID":"827af078-68a9-4e0e-92c1-dd7e6e337052","Type":"ContainerDied","Data":"7a82e6fbed030875f370c4cdf32867003c80232060b73c48c82aca534a36d2fa"} Jul 10 17:00:09.043245 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:09.043160 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a82e6fbed030875f370c4cdf32867003c80232060b73c48c82aca534a36d2fa" Jul 10 17:00:19.080727 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:19.080640 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:19.172717 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:19.172681 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:19.666793 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:19.666756 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:19.937862 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:19.937772 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:20.045470 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.045436 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:20.141754 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.141724 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl"] Jul 10 17:00:20.142122 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.142038 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827af078-68a9-4e0e-92c1-dd7e6e337052" containerName="pull" Jul 10 17:00:20.142122 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.142048 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="827af078-68a9-4e0e-92c1-dd7e6e337052" containerName="pull" Jul 10 17:00:20.142122 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.142059 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827af078-68a9-4e0e-92c1-dd7e6e337052" containerName="extract" Jul 10 17:00:20.142122 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.142064 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="827af078-68a9-4e0e-92c1-dd7e6e337052" containerName="extract" Jul 10 17:00:20.142122 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.142079 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827af078-68a9-4e0e-92c1-dd7e6e337052" containerName="util" Jul 10 17:00:20.142122 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.142085 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="827af078-68a9-4e0e-92c1-dd7e6e337052" containerName="util" Jul 10 17:00:20.142330 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.142133 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="827af078-68a9-4e0e-92c1-dd7e6e337052" containerName="extract" Jul 10 17:00:20.144319 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.144303 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.147074 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.147049 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 17:00:20.147283 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.147270 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 17:00:20.147349 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.147298 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gvgdz\"" Jul 10 17:00:20.147542 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.147521 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 17:00:20.147595 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.147573 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 17:00:20.163072 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.163040 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl"] Jul 10 17:00:20.255284 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.255208 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a2ff6c0-1e9c-418a-a34a-374fec422089-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.255284 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.255255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a2ff6c0-1e9c-418a-a34a-374fec422089-tmp\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.255476 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.255348 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpkgf\" (UniqueName: \"kubernetes.io/projected/0a2ff6c0-1e9c-418a-a34a-374fec422089-kube-api-access-kpkgf\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.255476 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.255385 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a2ff6c0-1e9c-418a-a34a-374fec422089-webhook-cert\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.255476 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.255407 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a2ff6c0-1e9c-418a-a34a-374fec422089-manifests\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.299232 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.299174 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:20.355949 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.355918 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpkgf\" (UniqueName: \"kubernetes.io/projected/0a2ff6c0-1e9c-418a-a34a-374fec422089-kube-api-access-kpkgf\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.356116 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.355969 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a2ff6c0-1e9c-418a-a34a-374fec422089-webhook-cert\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.356116 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.356000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a2ff6c0-1e9c-418a-a34a-374fec422089-manifests\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.356116 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.356047 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a2ff6c0-1e9c-418a-a34a-374fec422089-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.356116 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.356084 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a2ff6c0-1e9c-418a-a34a-374fec422089-tmp\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.356510 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.356471 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a2ff6c0-1e9c-418a-a34a-374fec422089-manifests\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.358881 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.358853 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a2ff6c0-1e9c-418a-a34a-374fec422089-tmp\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.359388 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.359370 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 17:00:20.367941 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.367863 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 17:00:20.369570 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.369485 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a2ff6c0-1e9c-418a-a34a-374fec422089-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.369887 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.369864 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a2ff6c0-1e9c-418a-a34a-374fec422089-webhook-cert\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.379372 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.379340 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 17:00:20.389964 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.389933 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpkgf\" (UniqueName: \"kubernetes.io/projected/0a2ff6c0-1e9c-418a-a34a-374fec422089-kube-api-access-kpkgf\") pod \"opendatahub-operator-controller-manager-67764b8b78-2vvvl\" (UID: \"0a2ff6c0-1e9c-418a-a34a-374fec422089\") " pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.402586 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.402544 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:20.458140 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.458114 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gvgdz\"" Jul 10 17:00:20.466312 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.466283 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:20.501068 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.501037 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:20.634404 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.634379 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl"] Jul 10 17:00:20.637087 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:00:20.637057 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2ff6c0_1e9c_418a_a34a_374fec422089.slice/crio-cfc6354ed5ba02a26d72afe017ad6e507fa0c4255b4263ad6ac1eec636419f2d WatchSource:0}: Error finding container cfc6354ed5ba02a26d72afe017ad6e507fa0c4255b4263ad6ac1eec636419f2d: Status 404 returned error can't find the container with id cfc6354ed5ba02a26d72afe017ad6e507fa0c4255b4263ad6ac1eec636419f2d Jul 10 17:00:20.996905 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:20.996865 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:21.092995 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:21.092967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:21.098134 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:21.098105 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" event={"ID":"0a2ff6c0-1e9c-418a-a34a-374fec422089","Type":"ContainerStarted","Data":"cfc6354ed5ba02a26d72afe017ad6e507fa0c4255b4263ad6ac1eec636419f2d"} Jul 10 17:00:24.115479 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:24.115435 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" event={"ID":"0a2ff6c0-1e9c-418a-a34a-374fec422089","Type":"ContainerStarted","Data":"2ce50b19ba5fd122d9caab50608f1f03ced36de445e19506df97686d8e72163c"} Jul 10 17:00:24.938478 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:24.938435 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:25.039599 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:25.039558 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:26.126301 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:26.126251 2559 generic.go:356] "Generic (PLEG): container finished" podID="0a2ff6c0-1e9c-418a-a34a-374fec422089" containerID="2ce50b19ba5fd122d9caab50608f1f03ced36de445e19506df97686d8e72163c" exitCode=0 Jul 10 17:00:26.126749 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:26.126321 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" event={"ID":"0a2ff6c0-1e9c-418a-a34a-374fec422089","Type":"ContainerDied","Data":"2ce50b19ba5fd122d9caab50608f1f03ced36de445e19506df97686d8e72163c"} Jul 10 17:00:27.133052 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:27.133015 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" event={"ID":"0a2ff6c0-1e9c-418a-a34a-374fec422089","Type":"ContainerStarted","Data":"3f696b078432fad3fd9d4487976ebeba3f1fa248442beb456c2517087051b996"} Jul 10 17:00:27.133440 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:27.133233 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:27.162603 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:27.162537 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" podStartSLOduration=2.21050982 podStartE2EDuration="7.16252002s" totalImagesPullingTime="3.776080256s" totalInitContainerRuntime="1.175929944s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:00:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:00:27.160885696 +0000 UTC m=+367.286594171" watchObservedRunningTime="2026-07-10 17:00:27.16252002 +0000 UTC m=+367.288228494" Jul 10 17:00:31.869277 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:31.869241 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:31.969939 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:31.969909 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:38.140384 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:38.140352 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67764b8b78-2vvvl" Jul 10 17:00:40.002072 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:40.002028 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:40.139268 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:40.139234 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:40.514347 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:40.514313 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:40.763312 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:40.763277 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:41.046662 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:41.046623 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:41.111921 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:41.111886 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:41.196802 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:41.196768 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:41.724680 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:41.724642 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:42.174700 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:42.174653 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:42.320335 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:42.320287 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:42.393572 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:42.393536 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:42.810674 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:42.810641 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:43.521117 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:43.521080 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:48.518463 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:48.518429 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:48.699524 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:48.699456 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:49.400828 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.400791 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl"] Jul 10 17:00:49.403184 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.403165 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.406088 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.406010 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 17:00:49.406557 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.406522 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 17:00:49.406557 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.406548 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 17:00:49.406718 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.406654 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jbhxs\"" Jul 10 17:00:49.415660 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.415617 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.415790 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.415687 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.415790 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.415774 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwncg\" (UniqueName: \"kubernetes.io/projected/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-kube-api-access-xwncg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.415902 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.415847 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.415902 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.415890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.416007 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.415920 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.416067 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.416027 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.416122 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.416072 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.416122 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.416115 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.416736 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.416716 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl"] Jul 10 17:00:49.447398 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.447364 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:49.517626 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.517588 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.517811 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.517635 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.517811 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.517681 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.517811 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.517727 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwncg\" (UniqueName: \"kubernetes.io/projected/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-kube-api-access-xwncg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.517811 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.517767 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.517811 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.517799 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.518063 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.517907 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.518063 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.517985 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.518063 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.518017 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.518191 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.518161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.518389 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.518363 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.518532 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.518402 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.518532 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.518419 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.518532 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.518487 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.520216 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.520194 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.520430 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.520413 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.541446 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.541418 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.541627 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.541578 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwncg\" (UniqueName: \"kubernetes.io/projected/9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd-kube-api-access-xwncg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jl2jl\" (UID: \"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.645723 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.645694 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:49.714466 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.714382 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:49.812554 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.812516 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:49.868236 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.868203 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl"] Jul 10 17:00:49.869934 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:00:49.869906 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1e3bab_dae9_4a2c_a6e9_a428cf3fc6bd.slice/crio-e078ccef458acfb8453994a65b3c01f4f2de0d107018d0cbfce85d14ea336b1f WatchSource:0}: Error finding container e078ccef458acfb8453994a65b3c01f4f2de0d107018d0cbfce85d14ea336b1f: Status 404 returned error can't find the container with id e078ccef458acfb8453994a65b3c01f4f2de0d107018d0cbfce85d14ea336b1f Jul 10 17:00:49.982123 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.982050 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s"] Jul 10 17:00:49.985396 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.985371 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:49.989578 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.989556 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 17:00:49.989718 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.989599 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2jxfb\"" Jul 10 17:00:49.989782 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.989713 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 17:00:49.989782 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:49.989776 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 17:00:50.005530 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.005485 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s"] Jul 10 17:00:50.022822 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.022788 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/019f277a-59b3-43f3-b73c-f049d9e812b3-manager-config\") pod \"lws-controller-manager-7bcc59c859-kfv6s\" (UID: \"019f277a-59b3-43f3-b73c-f049d9e812b3\") " pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:50.022986 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.022844 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/019f277a-59b3-43f3-b73c-f049d9e812b3-metrics-cert\") pod \"lws-controller-manager-7bcc59c859-kfv6s\" (UID: \"019f277a-59b3-43f3-b73c-f049d9e812b3\") " pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:50.022986 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.022948 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/019f277a-59b3-43f3-b73c-f049d9e812b3-cert\") pod \"lws-controller-manager-7bcc59c859-kfv6s\" (UID: \"019f277a-59b3-43f3-b73c-f049d9e812b3\") " pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:50.023061 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.022984 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c24jr\" (UniqueName: \"kubernetes.io/projected/019f277a-59b3-43f3-b73c-f049d9e812b3-kube-api-access-c24jr\") pod \"lws-controller-manager-7bcc59c859-kfv6s\" (UID: \"019f277a-59b3-43f3-b73c-f049d9e812b3\") " pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:50.124405 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.124363 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/019f277a-59b3-43f3-b73c-f049d9e812b3-manager-config\") pod \"lws-controller-manager-7bcc59c859-kfv6s\" (UID: \"019f277a-59b3-43f3-b73c-f049d9e812b3\") " pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:50.124583 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.124423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/019f277a-59b3-43f3-b73c-f049d9e812b3-metrics-cert\") pod \"lws-controller-manager-7bcc59c859-kfv6s\" (UID: \"019f277a-59b3-43f3-b73c-f049d9e812b3\") " pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:50.124583 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.124483 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/019f277a-59b3-43f3-b73c-f049d9e812b3-cert\") pod \"lws-controller-manager-7bcc59c859-kfv6s\" (UID: \"019f277a-59b3-43f3-b73c-f049d9e812b3\") " pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:50.124583 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.124541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c24jr\" (UniqueName: \"kubernetes.io/projected/019f277a-59b3-43f3-b73c-f049d9e812b3-kube-api-access-c24jr\") pod \"lws-controller-manager-7bcc59c859-kfv6s\" (UID: \"019f277a-59b3-43f3-b73c-f049d9e812b3\") " pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:50.125165 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.125135 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/019f277a-59b3-43f3-b73c-f049d9e812b3-manager-config\") pod \"lws-controller-manager-7bcc59c859-kfv6s\" (UID: \"019f277a-59b3-43f3-b73c-f049d9e812b3\") " pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:50.127119 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.127097 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/019f277a-59b3-43f3-b73c-f049d9e812b3-metrics-cert\") pod \"lws-controller-manager-7bcc59c859-kfv6s\" (UID: \"019f277a-59b3-43f3-b73c-f049d9e812b3\") " pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:50.127229 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.127171 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/019f277a-59b3-43f3-b73c-f049d9e812b3-cert\") pod \"lws-controller-manager-7bcc59c859-kfv6s\" (UID: \"019f277a-59b3-43f3-b73c-f049d9e812b3\") " pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:50.137278 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.137245 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c24jr\" (UniqueName: \"kubernetes.io/projected/019f277a-59b3-43f3-b73c-f049d9e812b3-kube-api-access-c24jr\") pod \"lws-controller-manager-7bcc59c859-kfv6s\" (UID: \"019f277a-59b3-43f3-b73c-f049d9e812b3\") " pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:50.260519 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.260398 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" event={"ID":"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd","Type":"ContainerStarted","Data":"e078ccef458acfb8453994a65b3c01f4f2de0d107018d0cbfce85d14ea336b1f"} Jul 10 17:00:50.294918 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.294880 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:50.466181 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:50.466154 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s"] Jul 10 17:00:50.468639 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:00:50.468614 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019f277a_59b3_43f3_b73c_f049d9e812b3.slice/crio-85501f345734e4f91c797d54cca535ff5e73d3c74ee6fcd1795547293bc03d32 WatchSource:0}: Error finding container 85501f345734e4f91c797d54cca535ff5e73d3c74ee6fcd1795547293bc03d32: Status 404 returned error can't find the container with id 85501f345734e4f91c797d54cca535ff5e73d3c74ee6fcd1795547293bc03d32 Jul 10 17:00:51.266246 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.266198 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" event={"ID":"019f277a-59b3-43f3-b73c-f049d9e812b3","Type":"ContainerStarted","Data":"85501f345734e4f91c797d54cca535ff5e73d3c74ee6fcd1795547293bc03d32"} Jul 10 17:00:51.594616 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.594585 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:51.748577 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.748543 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm"] Jul 10 17:00:51.750561 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.750527 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:51.754337 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.754307 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm" Jul 10 17:00:51.760230 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.760206 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 10 17:00:51.763444 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.763410 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8szmk\"" Jul 10 17:00:51.777131 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.777103 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm"] Jul 10 17:00:51.846785 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.846694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a41b3e8-d802-46aa-b2d1-685760fb4538-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v5nkm\" (UID: \"2a41b3e8-d802-46aa-b2d1-685760fb4538\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm" Jul 10 17:00:51.846785 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.846753 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc96h\" (UniqueName: \"kubernetes.io/projected/2a41b3e8-d802-46aa-b2d1-685760fb4538-kube-api-access-zc96h\") pod \"llmisvc-controller-manager-777c6bd7d5-v5nkm\" (UID: \"2a41b3e8-d802-46aa-b2d1-685760fb4538\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm" Jul 10 17:00:51.948142 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.948101 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc96h\" (UniqueName: \"kubernetes.io/projected/2a41b3e8-d802-46aa-b2d1-685760fb4538-kube-api-access-zc96h\") pod \"llmisvc-controller-manager-777c6bd7d5-v5nkm\" (UID: \"2a41b3e8-d802-46aa-b2d1-685760fb4538\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm" Jul 10 17:00:51.948333 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.948259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a41b3e8-d802-46aa-b2d1-685760fb4538-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v5nkm\" (UID: \"2a41b3e8-d802-46aa-b2d1-685760fb4538\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm" Jul 10 17:00:51.951710 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.951681 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a41b3e8-d802-46aa-b2d1-685760fb4538-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v5nkm\" (UID: \"2a41b3e8-d802-46aa-b2d1-685760fb4538\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm" Jul 10 17:00:51.982418 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:51.982383 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc96h\" (UniqueName: \"kubernetes.io/projected/2a41b3e8-d802-46aa-b2d1-685760fb4538-kube-api-access-zc96h\") pod \"llmisvc-controller-manager-777c6bd7d5-v5nkm\" (UID: \"2a41b3e8-d802-46aa-b2d1-685760fb4538\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm" Jul 10 17:00:52.068452 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:52.068421 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm" Jul 10 17:00:52.106110 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:52.106026 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:52.270872 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:52.270830 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:52.585808 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:52.585768 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:00:52.585959 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:52.585871 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:00:52.585959 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:52.585904 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:00:52.698238 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:52.698207 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm"] Jul 10 17:00:52.700694 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:00:52.700661 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2a41b3e8_d802_46aa_b2d1_685760fb4538.slice/crio-c6bade37fb25fa42ee82082423722f6ae2b52f1994a9d447ae34da713d21bc4a WatchSource:0}: Error finding container c6bade37fb25fa42ee82082423722f6ae2b52f1994a9d447ae34da713d21bc4a: Status 404 returned error can't find the container with id c6bade37fb25fa42ee82082423722f6ae2b52f1994a9d447ae34da713d21bc4a Jul 10 17:00:53.023680 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:53.023646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:53.118643 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:53.118611 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:00:53.281873 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:53.281772 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm" event={"ID":"2a41b3e8-d802-46aa-b2d1-685760fb4538","Type":"ContainerStarted","Data":"c6bade37fb25fa42ee82082423722f6ae2b52f1994a9d447ae34da713d21bc4a"} Jul 10 17:00:53.284083 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:53.284038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" event={"ID":"9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd","Type":"ContainerStarted","Data":"5194289b4fd63ccaa0de7f8c3de43bf3882a65dbf414b3ce2979af0c060f5212"} Jul 10 17:00:53.337521 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:53.337411 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" podStartSLOduration=1.6239057529999998 podStartE2EDuration="4.337368641s" totalImagesPullingTime="2.713462888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:00:53.33705581 +0000 UTC m=+393.462764221" watchObservedRunningTime="2026-07-10 17:00:53.337368641 +0000 UTC m=+393.463077061" Jul 10 17:00:53.715112 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:53.715051 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:53.720142 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:53.720113 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:54.295098 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:54.295001 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" event={"ID":"019f277a-59b3-43f3-b73c-f049d9e812b3","Type":"ContainerStarted","Data":"9973707e11c6e52c8b595134ee2872352cd963f1509e83eed09843e8ca6cb3af"} Jul 10 17:00:54.295732 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:54.295231 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:54.295732 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:54.295272 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:00:54.296361 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:54.296342 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jl2jl" Jul 10 17:00:54.322616 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:54.322552 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" podStartSLOduration=1.7886370600000001 podStartE2EDuration="5.322532857s" totalImagesPullingTime="3.533895797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:00:54.319177085 +0000 UTC m=+394.444885496" watchObservedRunningTime="2026-07-10 17:00:54.322532857 +0000 UTC m=+394.448241267" Jul 10 17:00:57.313029 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:57.312936 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm" event={"ID":"2a41b3e8-d802-46aa-b2d1-685760fb4538","Type":"ContainerStarted","Data":"8c61275e04bc86932e177df0125e58b5a1c697a587213000534b79bacfc838d5"} Jul 10 17:00:57.313395 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:57.313076 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm" Jul 10 17:00:57.347132 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:00:57.347083 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm" podStartSLOduration=2.028261297 podStartE2EDuration="6.347041534s" totalImagesPullingTime="4.318780237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:00:57.34517912 +0000 UTC m=+397.470887530" watchObservedRunningTime="2026-07-10 17:00:57.347041534 +0000 UTC m=+397.472749955" Jul 10 17:01:03.736321 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:03.736286 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:01:05.302468 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:05.302437 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7bcc59c859-kfv6s" Jul 10 17:01:28.322993 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:28.322959 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5nkm" Jul 10 17:01:41.579745 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:41.579625 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:01:41.674307 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:41.674275 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:01:41.774197 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:41.774164 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:01:42.513183 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:42.513148 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:01:42.723969 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:42.723927 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:01:57.116147 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.116107 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k"] Jul 10 17:01:57.119903 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.119885 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" Jul 10 17:01:57.122345 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.122322 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 17:01:57.122483 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.122354 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 17:01:57.123165 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.123149 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9g9z6\"" Jul 10 17:01:57.127979 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.127952 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k"] Jul 10 17:01:57.241848 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.241806 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgfbb\" (UniqueName: \"kubernetes.io/projected/86807469-b337-4585-a3fb-e47255061a96-kube-api-access-bgfbb\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k\" (UID: \"86807469-b337-4585-a3fb-e47255061a96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" Jul 10 17:01:57.242027 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.241867 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86807469-b337-4585-a3fb-e47255061a96-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k\" (UID: \"86807469-b337-4585-a3fb-e47255061a96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" Jul 10 17:01:57.242027 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.241922 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86807469-b337-4585-a3fb-e47255061a96-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k\" (UID: \"86807469-b337-4585-a3fb-e47255061a96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" Jul 10 17:01:57.342335 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.342277 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgfbb\" (UniqueName: \"kubernetes.io/projected/86807469-b337-4585-a3fb-e47255061a96-kube-api-access-bgfbb\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k\" (UID: \"86807469-b337-4585-a3fb-e47255061a96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" Jul 10 17:01:57.342574 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.342363 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86807469-b337-4585-a3fb-e47255061a96-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k\" (UID: \"86807469-b337-4585-a3fb-e47255061a96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" Jul 10 17:01:57.342574 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.342399 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86807469-b337-4585-a3fb-e47255061a96-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k\" (UID: \"86807469-b337-4585-a3fb-e47255061a96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" Jul 10 17:01:57.342764 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.342744 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86807469-b337-4585-a3fb-e47255061a96-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k\" (UID: \"86807469-b337-4585-a3fb-e47255061a96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" Jul 10 17:01:57.342850 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.342824 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86807469-b337-4585-a3fb-e47255061a96-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k\" (UID: \"86807469-b337-4585-a3fb-e47255061a96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" Jul 10 17:01:57.351363 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.351334 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgfbb\" (UniqueName: \"kubernetes.io/projected/86807469-b337-4585-a3fb-e47255061a96-kube-api-access-bgfbb\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k\" (UID: \"86807469-b337-4585-a3fb-e47255061a96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" Jul 10 17:01:57.430553 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.430442 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" Jul 10 17:01:57.565694 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.565671 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k"] Jul 10 17:01:57.567564 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:01:57.567528 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86807469_b337_4585_a3fb_e47255061a96.slice/crio-66b4d117adf0c141f91f5063bd1cabab4035340f1cff00545f9b0f1820e4ee1d WatchSource:0}: Error finding container 66b4d117adf0c141f91f5063bd1cabab4035340f1cff00545f9b0f1820e4ee1d: Status 404 returned error can't find the container with id 66b4d117adf0c141f91f5063bd1cabab4035340f1cff00545f9b0f1820e4ee1d Jul 10 17:01:57.605752 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.605717 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" event={"ID":"86807469-b337-4585-a3fb-e47255061a96","Type":"ContainerStarted","Data":"66b4d117adf0c141f91f5063bd1cabab4035340f1cff00545f9b0f1820e4ee1d"} Jul 10 17:01:57.727261 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.727230 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk"] Jul 10 17:01:57.730952 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.730926 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" Jul 10 17:01:57.740760 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.740728 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk"] Jul 10 17:01:57.845520 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.845439 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f300a123-c7ac-4314-84d2-5d6c02df5300-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk\" (UID: \"f300a123-c7ac-4314-84d2-5d6c02df5300\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" Jul 10 17:01:57.845710 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.845585 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f300a123-c7ac-4314-84d2-5d6c02df5300-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk\" (UID: \"f300a123-c7ac-4314-84d2-5d6c02df5300\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" Jul 10 17:01:57.845710 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.845616 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmrns\" (UniqueName: \"kubernetes.io/projected/f300a123-c7ac-4314-84d2-5d6c02df5300-kube-api-access-jmrns\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk\" (UID: \"f300a123-c7ac-4314-84d2-5d6c02df5300\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" Jul 10 17:01:57.946681 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.946597 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f300a123-c7ac-4314-84d2-5d6c02df5300-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk\" (UID: \"f300a123-c7ac-4314-84d2-5d6c02df5300\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" Jul 10 17:01:57.946681 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.946647 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrns\" (UniqueName: \"kubernetes.io/projected/f300a123-c7ac-4314-84d2-5d6c02df5300-kube-api-access-jmrns\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk\" (UID: \"f300a123-c7ac-4314-84d2-5d6c02df5300\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" Jul 10 17:01:57.946831 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.946716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f300a123-c7ac-4314-84d2-5d6c02df5300-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk\" (UID: \"f300a123-c7ac-4314-84d2-5d6c02df5300\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" Jul 10 17:01:57.947016 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.946993 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f300a123-c7ac-4314-84d2-5d6c02df5300-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk\" (UID: \"f300a123-c7ac-4314-84d2-5d6c02df5300\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" Jul 10 17:01:57.947085 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.947064 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f300a123-c7ac-4314-84d2-5d6c02df5300-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk\" (UID: \"f300a123-c7ac-4314-84d2-5d6c02df5300\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" Jul 10 17:01:57.956298 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:57.956271 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrns\" (UniqueName: \"kubernetes.io/projected/f300a123-c7ac-4314-84d2-5d6c02df5300-kube-api-access-jmrns\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk\" (UID: \"f300a123-c7ac-4314-84d2-5d6c02df5300\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" Jul 10 17:01:58.043617 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.043578 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" Jul 10 17:01:58.125445 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.125409 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg"] Jul 10 17:01:58.129333 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.129313 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" Jul 10 17:01:58.139607 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.138378 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg"] Jul 10 17:01:58.187627 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.184435 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk"] Jul 10 17:01:58.249175 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.249144 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8x6\" (UniqueName: \"kubernetes.io/projected/9431c544-e4c6-4387-8808-f98aeb42ab81-kube-api-access-bw8x6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg\" (UID: \"9431c544-e4c6-4387-8808-f98aeb42ab81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" Jul 10 17:01:58.249291 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.249193 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9431c544-e4c6-4387-8808-f98aeb42ab81-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg\" (UID: \"9431c544-e4c6-4387-8808-f98aeb42ab81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" Jul 10 17:01:58.249291 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.249260 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9431c544-e4c6-4387-8808-f98aeb42ab81-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg\" (UID: \"9431c544-e4c6-4387-8808-f98aeb42ab81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" Jul 10 17:01:58.350546 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.350475 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9431c544-e4c6-4387-8808-f98aeb42ab81-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg\" (UID: \"9431c544-e4c6-4387-8808-f98aeb42ab81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" Jul 10 17:01:58.350736 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.350596 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9431c544-e4c6-4387-8808-f98aeb42ab81-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg\" (UID: \"9431c544-e4c6-4387-8808-f98aeb42ab81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" Jul 10 17:01:58.350736 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.350683 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8x6\" (UniqueName: \"kubernetes.io/projected/9431c544-e4c6-4387-8808-f98aeb42ab81-kube-api-access-bw8x6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg\" (UID: \"9431c544-e4c6-4387-8808-f98aeb42ab81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" Jul 10 17:01:58.350973 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.350938 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9431c544-e4c6-4387-8808-f98aeb42ab81-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg\" (UID: \"9431c544-e4c6-4387-8808-f98aeb42ab81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" Jul 10 17:01:58.350973 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.350955 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9431c544-e4c6-4387-8808-f98aeb42ab81-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg\" (UID: \"9431c544-e4c6-4387-8808-f98aeb42ab81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" Jul 10 17:01:58.360440 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.360409 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8x6\" (UniqueName: \"kubernetes.io/projected/9431c544-e4c6-4387-8808-f98aeb42ab81-kube-api-access-bw8x6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg\" (UID: \"9431c544-e4c6-4387-8808-f98aeb42ab81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" Jul 10 17:01:58.442259 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.442226 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" Jul 10 17:01:58.515649 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.515616 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x"] Jul 10 17:01:58.519704 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.519667 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" Jul 10 17:01:58.528196 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.528165 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x"] Jul 10 17:01:58.586190 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.586166 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg"] Jul 10 17:01:58.589201 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:01:58.589161 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9431c544_e4c6_4387_8808_f98aeb42ab81.slice/crio-c32e13fd79f141853ac454435b124c460f3f83ec0766221e563f26e4391b215c WatchSource:0}: Error finding container c32e13fd79f141853ac454435b124c460f3f83ec0766221e563f26e4391b215c: Status 404 returned error can't find the container with id c32e13fd79f141853ac454435b124c460f3f83ec0766221e563f26e4391b215c Jul 10 17:01:58.612172 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.612139 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" event={"ID":"9431c544-e4c6-4387-8808-f98aeb42ab81","Type":"ContainerStarted","Data":"c32e13fd79f141853ac454435b124c460f3f83ec0766221e563f26e4391b215c"} Jul 10 17:01:58.613566 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.613543 2559 generic.go:356] "Generic (PLEG): container finished" podID="f300a123-c7ac-4314-84d2-5d6c02df5300" containerID="41a1e8b193099ce443e26d8dd0187c74c179f74beef3eef24facef16dae81aea" exitCode=0 Jul 10 17:01:58.613674 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.613630 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" event={"ID":"f300a123-c7ac-4314-84d2-5d6c02df5300","Type":"ContainerDied","Data":"41a1e8b193099ce443e26d8dd0187c74c179f74beef3eef24facef16dae81aea"} Jul 10 17:01:58.613735 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.613677 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" event={"ID":"f300a123-c7ac-4314-84d2-5d6c02df5300","Type":"ContainerStarted","Data":"b9c377a9b6c66c7ccefd94c0b1ce017b097535a3f4aa6c1d66d193e08294cf3c"} Jul 10 17:01:58.615140 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.615119 2559 generic.go:356] "Generic (PLEG): container finished" podID="86807469-b337-4585-a3fb-e47255061a96" containerID="473a89f3ad0d24bfcab442a7de9a58473671ffe37582bcf547d56c260d0fbbec" exitCode=0 Jul 10 17:01:58.615243 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.615195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" event={"ID":"86807469-b337-4585-a3fb-e47255061a96","Type":"ContainerDied","Data":"473a89f3ad0d24bfcab442a7de9a58473671ffe37582bcf547d56c260d0fbbec"} Jul 10 17:01:58.653365 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.653337 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d271dec-621f-4218-86d1-1143a6f60558-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x\" (UID: \"5d271dec-621f-4218-86d1-1143a6f60558\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" Jul 10 17:01:58.653543 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.653526 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d271dec-621f-4218-86d1-1143a6f60558-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x\" (UID: \"5d271dec-621f-4218-86d1-1143a6f60558\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" Jul 10 17:01:58.653607 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.653553 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqqbw\" (UniqueName: \"kubernetes.io/projected/5d271dec-621f-4218-86d1-1143a6f60558-kube-api-access-sqqbw\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x\" (UID: \"5d271dec-621f-4218-86d1-1143a6f60558\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" Jul 10 17:01:58.754287 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.754191 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d271dec-621f-4218-86d1-1143a6f60558-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x\" (UID: \"5d271dec-621f-4218-86d1-1143a6f60558\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" Jul 10 17:01:58.754287 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.754226 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqqbw\" (UniqueName: \"kubernetes.io/projected/5d271dec-621f-4218-86d1-1143a6f60558-kube-api-access-sqqbw\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x\" (UID: \"5d271dec-621f-4218-86d1-1143a6f60558\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" Jul 10 17:01:58.754287 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.754267 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d271dec-621f-4218-86d1-1143a6f60558-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x\" (UID: \"5d271dec-621f-4218-86d1-1143a6f60558\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" Jul 10 17:01:58.754669 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.754643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d271dec-621f-4218-86d1-1143a6f60558-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x\" (UID: \"5d271dec-621f-4218-86d1-1143a6f60558\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" Jul 10 17:01:58.754718 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.754660 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d271dec-621f-4218-86d1-1143a6f60558-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x\" (UID: \"5d271dec-621f-4218-86d1-1143a6f60558\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" Jul 10 17:01:58.763049 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.763029 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqqbw\" (UniqueName: \"kubernetes.io/projected/5d271dec-621f-4218-86d1-1143a6f60558-kube-api-access-sqqbw\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x\" (UID: \"5d271dec-621f-4218-86d1-1143a6f60558\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" Jul 10 17:01:58.831818 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.831782 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" Jul 10 17:01:58.966253 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:58.966147 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x"] Jul 10 17:01:58.969011 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:01:58.968986 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d271dec_621f_4218_86d1_1143a6f60558.slice/crio-6d14f1b017461e712803acec0e6ed5cfa405180272ceb9a09ec463f53d2001a3 WatchSource:0}: Error finding container 6d14f1b017461e712803acec0e6ed5cfa405180272ceb9a09ec463f53d2001a3: Status 404 returned error can't find the container with id 6d14f1b017461e712803acec0e6ed5cfa405180272ceb9a09ec463f53d2001a3 Jul 10 17:01:59.623541 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:59.623506 2559 generic.go:356] "Generic (PLEG): container finished" podID="9431c544-e4c6-4387-8808-f98aeb42ab81" containerID="81a6faba8db63071c3c0d3b16783460b059029f830d6d81d1226487fa2f08de0" exitCode=0 Jul 10 17:01:59.623541 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:59.623526 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" event={"ID":"9431c544-e4c6-4387-8808-f98aeb42ab81","Type":"ContainerDied","Data":"81a6faba8db63071c3c0d3b16783460b059029f830d6d81d1226487fa2f08de0"} Jul 10 17:01:59.625093 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:59.625070 2559 generic.go:356] "Generic (PLEG): container finished" podID="5d271dec-621f-4218-86d1-1143a6f60558" containerID="953f52c267b763d7c51668a3fa33cc0c46ab1e30778a795ee258ebefbffa048c" exitCode=0 Jul 10 17:01:59.625194 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:59.625156 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" event={"ID":"5d271dec-621f-4218-86d1-1143a6f60558","Type":"ContainerDied","Data":"953f52c267b763d7c51668a3fa33cc0c46ab1e30778a795ee258ebefbffa048c"} Jul 10 17:01:59.625249 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:59.625219 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" event={"ID":"5d271dec-621f-4218-86d1-1143a6f60558","Type":"ContainerStarted","Data":"6d14f1b017461e712803acec0e6ed5cfa405180272ceb9a09ec463f53d2001a3"} Jul 10 17:01:59.627294 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:59.627241 2559 generic.go:356] "Generic (PLEG): container finished" podID="f300a123-c7ac-4314-84d2-5d6c02df5300" containerID="ff9ff29761f7f40f05101776209e59142478a7e52f1eb0cc537fb6e762f2e5bf" exitCode=0 Jul 10 17:01:59.627393 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:01:59.627320 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" event={"ID":"f300a123-c7ac-4314-84d2-5d6c02df5300","Type":"ContainerDied","Data":"ff9ff29761f7f40f05101776209e59142478a7e52f1eb0cc537fb6e762f2e5bf"} Jul 10 17:02:00.634274 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:00.634239 2559 generic.go:356] "Generic (PLEG): container finished" podID="9431c544-e4c6-4387-8808-f98aeb42ab81" containerID="8260cf32750fbd2e691164ed8744a39afac9cfbdfaf6aed2d621df1ca2ebc787" exitCode=0 Jul 10 17:02:00.634739 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:00.634302 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" event={"ID":"9431c544-e4c6-4387-8808-f98aeb42ab81","Type":"ContainerDied","Data":"8260cf32750fbd2e691164ed8744a39afac9cfbdfaf6aed2d621df1ca2ebc787"} Jul 10 17:02:00.636623 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:00.636599 2559 generic.go:356] "Generic (PLEG): container finished" podID="f300a123-c7ac-4314-84d2-5d6c02df5300" containerID="07b7c8398d58913c35e60b3f94328a2c13a0b85e5ecda5710559cfd6322e41b3" exitCode=0 Jul 10 17:02:00.636742 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:00.636693 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" event={"ID":"f300a123-c7ac-4314-84d2-5d6c02df5300","Type":"ContainerDied","Data":"07b7c8398d58913c35e60b3f94328a2c13a0b85e5ecda5710559cfd6322e41b3"} Jul 10 17:02:00.638412 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:00.638391 2559 generic.go:356] "Generic (PLEG): container finished" podID="86807469-b337-4585-a3fb-e47255061a96" containerID="130722c68a182090c186ed61561839f735d39e65628d0c87878b9a04e3e05f40" exitCode=0 Jul 10 17:02:00.638523 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:00.638470 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" event={"ID":"86807469-b337-4585-a3fb-e47255061a96","Type":"ContainerDied","Data":"130722c68a182090c186ed61561839f735d39e65628d0c87878b9a04e3e05f40"} Jul 10 17:02:00.761792 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:00.761759 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:01.645996 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.645905 2559 generic.go:356] "Generic (PLEG): container finished" podID="9431c544-e4c6-4387-8808-f98aeb42ab81" containerID="0ec4cceb5ef50888a44bd1672072af5fd07338e064cab97bb40743b802134564" exitCode=0 Jul 10 17:02:01.645996 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.645986 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" event={"ID":"9431c544-e4c6-4387-8808-f98aeb42ab81","Type":"ContainerDied","Data":"0ec4cceb5ef50888a44bd1672072af5fd07338e064cab97bb40743b802134564"} Jul 10 17:02:01.647730 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.647711 2559 generic.go:356] "Generic (PLEG): container finished" podID="5d271dec-621f-4218-86d1-1143a6f60558" containerID="baee5d8d017975f155752be5bdd9a5dfe091e5b1ee505962574f0e1ddcd65e39" exitCode=0 Jul 10 17:02:01.647828 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.647771 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" event={"ID":"5d271dec-621f-4218-86d1-1143a6f60558","Type":"ContainerDied","Data":"baee5d8d017975f155752be5bdd9a5dfe091e5b1ee505962574f0e1ddcd65e39"} Jul 10 17:02:01.649968 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.649944 2559 generic.go:356] "Generic (PLEG): container finished" podID="86807469-b337-4585-a3fb-e47255061a96" containerID="a9c8b6bf27e6edbaf66fe615fefd8affa4b9c23c5ca14b9cbc04333578e6686a" exitCode=0 Jul 10 17:02:01.650073 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.650040 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" event={"ID":"86807469-b337-4585-a3fb-e47255061a96","Type":"ContainerDied","Data":"a9c8b6bf27e6edbaf66fe615fefd8affa4b9c23c5ca14b9cbc04333578e6686a"} Jul 10 17:02:01.785390 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.785363 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" Jul 10 17:02:01.882850 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.882815 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f300a123-c7ac-4314-84d2-5d6c02df5300-util\") pod \"f300a123-c7ac-4314-84d2-5d6c02df5300\" (UID: \"f300a123-c7ac-4314-84d2-5d6c02df5300\") " Jul 10 17:02:01.883031 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.882873 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f300a123-c7ac-4314-84d2-5d6c02df5300-bundle\") pod \"f300a123-c7ac-4314-84d2-5d6c02df5300\" (UID: \"f300a123-c7ac-4314-84d2-5d6c02df5300\") " Jul 10 17:02:01.883031 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.882942 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmrns\" (UniqueName: \"kubernetes.io/projected/f300a123-c7ac-4314-84d2-5d6c02df5300-kube-api-access-jmrns\") pod \"f300a123-c7ac-4314-84d2-5d6c02df5300\" (UID: \"f300a123-c7ac-4314-84d2-5d6c02df5300\") " Jul 10 17:02:01.883550 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.883485 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f300a123-c7ac-4314-84d2-5d6c02df5300-bundle" (OuterVolumeSpecName: "bundle") pod "f300a123-c7ac-4314-84d2-5d6c02df5300" (UID: "f300a123-c7ac-4314-84d2-5d6c02df5300"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:02:01.885359 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.885331 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f300a123-c7ac-4314-84d2-5d6c02df5300-kube-api-access-jmrns" (OuterVolumeSpecName: "kube-api-access-jmrns") pod "f300a123-c7ac-4314-84d2-5d6c02df5300" (UID: "f300a123-c7ac-4314-84d2-5d6c02df5300"). InnerVolumeSpecName "kube-api-access-jmrns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:02:01.887793 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.887754 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f300a123-c7ac-4314-84d2-5d6c02df5300-util" (OuterVolumeSpecName: "util") pod "f300a123-c7ac-4314-84d2-5d6c02df5300" (UID: "f300a123-c7ac-4314-84d2-5d6c02df5300"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:02:01.984184 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.984084 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmrns\" (UniqueName: \"kubernetes.io/projected/f300a123-c7ac-4314-84d2-5d6c02df5300-kube-api-access-jmrns\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:02:01.984184 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.984117 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f300a123-c7ac-4314-84d2-5d6c02df5300-util\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:02:01.984184 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:01.984126 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f300a123-c7ac-4314-84d2-5d6c02df5300-bundle\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:02:02.656837 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.656755 2559 generic.go:356] "Generic (PLEG): container finished" podID="5d271dec-621f-4218-86d1-1143a6f60558" containerID="fe3ba5742416d65aca21d5332cc91f9a077aae26097b4e548e21340f655fd98d" exitCode=0 Jul 10 17:02:02.657274 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.656841 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" event={"ID":"5d271dec-621f-4218-86d1-1143a6f60558","Type":"ContainerDied","Data":"fe3ba5742416d65aca21d5332cc91f9a077aae26097b4e548e21340f655fd98d"} Jul 10 17:02:02.658812 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.658789 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" Jul 10 17:02:02.658812 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.658802 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk" event={"ID":"f300a123-c7ac-4314-84d2-5d6c02df5300","Type":"ContainerDied","Data":"b9c377a9b6c66c7ccefd94c0b1ce017b097535a3f4aa6c1d66d193e08294cf3c"} Jul 10 17:02:02.658943 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.658847 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9c377a9b6c66c7ccefd94c0b1ce017b097535a3f4aa6c1d66d193e08294cf3c" Jul 10 17:02:02.817248 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.817223 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" Jul 10 17:02:02.847560 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.847533 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" Jul 10 17:02:02.993201 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.993114 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9431c544-e4c6-4387-8808-f98aeb42ab81-util\") pod \"9431c544-e4c6-4387-8808-f98aeb42ab81\" (UID: \"9431c544-e4c6-4387-8808-f98aeb42ab81\") " Jul 10 17:02:02.993374 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.993212 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw8x6\" (UniqueName: \"kubernetes.io/projected/9431c544-e4c6-4387-8808-f98aeb42ab81-kube-api-access-bw8x6\") pod \"9431c544-e4c6-4387-8808-f98aeb42ab81\" (UID: \"9431c544-e4c6-4387-8808-f98aeb42ab81\") " Jul 10 17:02:02.993374 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.993265 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86807469-b337-4585-a3fb-e47255061a96-util\") pod \"86807469-b337-4585-a3fb-e47255061a96\" (UID: \"86807469-b337-4585-a3fb-e47255061a96\") " Jul 10 17:02:02.993374 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.993292 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86807469-b337-4585-a3fb-e47255061a96-bundle\") pod \"86807469-b337-4585-a3fb-e47255061a96\" (UID: \"86807469-b337-4585-a3fb-e47255061a96\") " Jul 10 17:02:02.993374 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.993331 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9431c544-e4c6-4387-8808-f98aeb42ab81-bundle\") pod \"9431c544-e4c6-4387-8808-f98aeb42ab81\" (UID: \"9431c544-e4c6-4387-8808-f98aeb42ab81\") " Jul 10 17:02:02.993679 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.993391 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgfbb\" (UniqueName: \"kubernetes.io/projected/86807469-b337-4585-a3fb-e47255061a96-kube-api-access-bgfbb\") pod \"86807469-b337-4585-a3fb-e47255061a96\" (UID: \"86807469-b337-4585-a3fb-e47255061a96\") " Jul 10 17:02:02.993918 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.993884 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86807469-b337-4585-a3fb-e47255061a96-bundle" (OuterVolumeSpecName: "bundle") pod "86807469-b337-4585-a3fb-e47255061a96" (UID: "86807469-b337-4585-a3fb-e47255061a96"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:02:02.993918 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.993902 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9431c544-e4c6-4387-8808-f98aeb42ab81-bundle" (OuterVolumeSpecName: "bundle") pod "9431c544-e4c6-4387-8808-f98aeb42ab81" (UID: "9431c544-e4c6-4387-8808-f98aeb42ab81"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:02:02.995595 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.995570 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9431c544-e4c6-4387-8808-f98aeb42ab81-kube-api-access-bw8x6" (OuterVolumeSpecName: "kube-api-access-bw8x6") pod "9431c544-e4c6-4387-8808-f98aeb42ab81" (UID: "9431c544-e4c6-4387-8808-f98aeb42ab81"). InnerVolumeSpecName "kube-api-access-bw8x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:02:02.996050 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.996024 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86807469-b337-4585-a3fb-e47255061a96-kube-api-access-bgfbb" (OuterVolumeSpecName: "kube-api-access-bgfbb") pod "86807469-b337-4585-a3fb-e47255061a96" (UID: "86807469-b337-4585-a3fb-e47255061a96"). InnerVolumeSpecName "kube-api-access-bgfbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:02:02.999058 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.999034 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86807469-b337-4585-a3fb-e47255061a96-util" (OuterVolumeSpecName: "util") pod "86807469-b337-4585-a3fb-e47255061a96" (UID: "86807469-b337-4585-a3fb-e47255061a96"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:02:02.999297 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:02.999278 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9431c544-e4c6-4387-8808-f98aeb42ab81-util" (OuterVolumeSpecName: "util") pod "9431c544-e4c6-4387-8808-f98aeb42ab81" (UID: "9431c544-e4c6-4387-8808-f98aeb42ab81"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:02:03.095100 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.095062 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgfbb\" (UniqueName: \"kubernetes.io/projected/86807469-b337-4585-a3fb-e47255061a96-kube-api-access-bgfbb\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:02:03.095100 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.095093 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9431c544-e4c6-4387-8808-f98aeb42ab81-util\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:02:03.095100 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.095106 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw8x6\" (UniqueName: \"kubernetes.io/projected/9431c544-e4c6-4387-8808-f98aeb42ab81-kube-api-access-bw8x6\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:02:03.095326 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.095117 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86807469-b337-4585-a3fb-e47255061a96-util\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:02:03.095326 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.095128 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86807469-b337-4585-a3fb-e47255061a96-bundle\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:02:03.095326 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.095137 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9431c544-e4c6-4387-8808-f98aeb42ab81-bundle\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:02:03.667927 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.667842 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" Jul 10 17:02:03.667927 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.667859 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg" event={"ID":"9431c544-e4c6-4387-8808-f98aeb42ab81","Type":"ContainerDied","Data":"c32e13fd79f141853ac454435b124c460f3f83ec0766221e563f26e4391b215c"} Jul 10 17:02:03.668350 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.667938 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c32e13fd79f141853ac454435b124c460f3f83ec0766221e563f26e4391b215c" Jul 10 17:02:03.669823 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.669803 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" Jul 10 17:02:03.669933 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.669856 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k" event={"ID":"86807469-b337-4585-a3fb-e47255061a96","Type":"ContainerDied","Data":"66b4d117adf0c141f91f5063bd1cabab4035340f1cff00545f9b0f1820e4ee1d"} Jul 10 17:02:03.669933 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.669906 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66b4d117adf0c141f91f5063bd1cabab4035340f1cff00545f9b0f1820e4ee1d" Jul 10 17:02:03.813370 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.813345 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" Jul 10 17:02:03.902421 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.902386 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d271dec-621f-4218-86d1-1143a6f60558-util\") pod \"5d271dec-621f-4218-86d1-1143a6f60558\" (UID: \"5d271dec-621f-4218-86d1-1143a6f60558\") " Jul 10 17:02:03.902619 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.902456 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqqbw\" (UniqueName: \"kubernetes.io/projected/5d271dec-621f-4218-86d1-1143a6f60558-kube-api-access-sqqbw\") pod \"5d271dec-621f-4218-86d1-1143a6f60558\" (UID: \"5d271dec-621f-4218-86d1-1143a6f60558\") " Jul 10 17:02:03.902619 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.902535 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d271dec-621f-4218-86d1-1143a6f60558-bundle\") pod \"5d271dec-621f-4218-86d1-1143a6f60558\" (UID: \"5d271dec-621f-4218-86d1-1143a6f60558\") " Jul 10 17:02:03.902993 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.902967 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d271dec-621f-4218-86d1-1143a6f60558-bundle" (OuterVolumeSpecName: "bundle") pod "5d271dec-621f-4218-86d1-1143a6f60558" (UID: "5d271dec-621f-4218-86d1-1143a6f60558"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:02:03.904750 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.904724 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d271dec-621f-4218-86d1-1143a6f60558-kube-api-access-sqqbw" (OuterVolumeSpecName: "kube-api-access-sqqbw") pod "5d271dec-621f-4218-86d1-1143a6f60558" (UID: "5d271dec-621f-4218-86d1-1143a6f60558"). InnerVolumeSpecName "kube-api-access-sqqbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:02:03.907913 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:03.907877 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d271dec-621f-4218-86d1-1143a6f60558-util" (OuterVolumeSpecName: "util") pod "5d271dec-621f-4218-86d1-1143a6f60558" (UID: "5d271dec-621f-4218-86d1-1143a6f60558"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:02:04.003921 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:04.003836 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d271dec-621f-4218-86d1-1143a6f60558-util\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:02:04.003921 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:04.003868 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqqbw\" (UniqueName: \"kubernetes.io/projected/5d271dec-621f-4218-86d1-1143a6f60558-kube-api-access-sqqbw\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:02:04.003921 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:04.003879 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d271dec-621f-4218-86d1-1143a6f60558-bundle\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:02:04.677306 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:04.677224 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" Jul 10 17:02:04.677306 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:04.677235 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x" event={"ID":"5d271dec-621f-4218-86d1-1143a6f60558","Type":"ContainerDied","Data":"6d14f1b017461e712803acec0e6ed5cfa405180272ceb9a09ec463f53d2001a3"} Jul 10 17:02:04.677306 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:04.677281 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d14f1b017461e712803acec0e6ed5cfa405180272ceb9a09ec463f53d2001a3" Jul 10 17:02:10.115556 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:10.115517 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:11.279448 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:11.279407 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:13.967544 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:13.967511 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:15.819111 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:15.819044 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:17.342879 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:17.342839 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:17.749043 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:17.748936 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:19.186228 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:19.186189 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:20.742895 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:20.742830 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:21.565231 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:21.565165 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:21.823287 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:21.823183 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:21.926981 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:21.926943 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:22.022666 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:22.022608 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:22.124363 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:22.124323 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:23.628158 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:23.628128 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:25.535459 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:25.535422 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:25.633636 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:25.633602 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:26.531240 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:26.531204 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:26.619010 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:26.618975 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:33.200288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:33.200252 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:33.302307 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:33.302276 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:36.646852 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.646817 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2lfw"] Jul 10 17:02:36.647488 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647457 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f300a123-c7ac-4314-84d2-5d6c02df5300" containerName="util" Jul 10 17:02:36.647651 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647486 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f300a123-c7ac-4314-84d2-5d6c02df5300" containerName="util" Jul 10 17:02:36.647651 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647531 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86807469-b337-4585-a3fb-e47255061a96" containerName="util" Jul 10 17:02:36.647651 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647540 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="86807469-b337-4585-a3fb-e47255061a96" containerName="util" Jul 10 17:02:36.647651 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647560 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86807469-b337-4585-a3fb-e47255061a96" containerName="extract" Jul 10 17:02:36.647651 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647568 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="86807469-b337-4585-a3fb-e47255061a96" containerName="extract" Jul 10 17:02:36.647651 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647583 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d271dec-621f-4218-86d1-1143a6f60558" containerName="util" Jul 10 17:02:36.647651 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647590 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d271dec-621f-4218-86d1-1143a6f60558" containerName="util" Jul 10 17:02:36.647651 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647599 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86807469-b337-4585-a3fb-e47255061a96" containerName="pull" Jul 10 17:02:36.647651 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647607 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="86807469-b337-4585-a3fb-e47255061a96" containerName="pull" Jul 10 17:02:36.647651 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647618 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9431c544-e4c6-4387-8808-f98aeb42ab81" containerName="extract" Jul 10 17:02:36.647651 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647625 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9431c544-e4c6-4387-8808-f98aeb42ab81" containerName="extract" Jul 10 17:02:36.647651 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647635 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d271dec-621f-4218-86d1-1143a6f60558" containerName="extract" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647666 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d271dec-621f-4218-86d1-1143a6f60558" containerName="extract" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647681 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9431c544-e4c6-4387-8808-f98aeb42ab81" containerName="util" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647686 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9431c544-e4c6-4387-8808-f98aeb42ab81" containerName="util" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647702 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f300a123-c7ac-4314-84d2-5d6c02df5300" containerName="extract" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647709 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f300a123-c7ac-4314-84d2-5d6c02df5300" containerName="extract" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647721 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9431c544-e4c6-4387-8808-f98aeb42ab81" containerName="pull" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647726 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9431c544-e4c6-4387-8808-f98aeb42ab81" containerName="pull" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647733 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f300a123-c7ac-4314-84d2-5d6c02df5300" containerName="pull" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647737 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f300a123-c7ac-4314-84d2-5d6c02df5300" containerName="pull" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647743 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d271dec-621f-4218-86d1-1143a6f60558" containerName="pull" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647747 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d271dec-621f-4218-86d1-1143a6f60558" containerName="pull" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647823 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9431c544-e4c6-4387-8808-f98aeb42ab81" containerName="extract" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647831 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="86807469-b337-4585-a3fb-e47255061a96" containerName="extract" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647837 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d271dec-621f-4218-86d1-1143a6f60558" containerName="extract" Jul 10 17:02:36.648288 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.647844 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f300a123-c7ac-4314-84d2-5d6c02df5300" containerName="extract" Jul 10 17:02:36.652577 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.652555 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2lfw" Jul 10 17:02:36.655525 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.655471 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 17:02:36.656596 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.656578 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 17:02:36.656692 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.656579 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-h42sg\"" Jul 10 17:02:36.670356 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.670327 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2lfw"] Jul 10 17:02:36.789065 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.789023 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79c9k\" (UniqueName: \"kubernetes.io/projected/04084f37-a997-436d-9496-49d7f1ee8fcc-kube-api-access-79c9k\") pod \"limitador-operator-controller-manager-859d7b78bd-h2lfw\" (UID: \"04084f37-a997-436d-9496-49d7f1ee8fcc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2lfw" Jul 10 17:02:36.889686 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.889641 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79c9k\" (UniqueName: \"kubernetes.io/projected/04084f37-a997-436d-9496-49d7f1ee8fcc-kube-api-access-79c9k\") pod \"limitador-operator-controller-manager-859d7b78bd-h2lfw\" (UID: \"04084f37-a997-436d-9496-49d7f1ee8fcc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2lfw" Jul 10 17:02:36.899923 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.899857 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79c9k\" (UniqueName: \"kubernetes.io/projected/04084f37-a997-436d-9496-49d7f1ee8fcc-kube-api-access-79c9k\") pod \"limitador-operator-controller-manager-859d7b78bd-h2lfw\" (UID: \"04084f37-a997-436d-9496-49d7f1ee8fcc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2lfw" Jul 10 17:02:36.963540 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:36.963456 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2lfw" Jul 10 17:02:37.117262 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:37.117213 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2lfw"] Jul 10 17:02:37.120195 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:02:37.120167 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04084f37_a997_436d_9496_49d7f1ee8fcc.slice/crio-1bd242d80ee09c9e02015b09d20ace04304335cd528758d2e232ab02c535ffb0 WatchSource:0}: Error finding container 1bd242d80ee09c9e02015b09d20ace04304335cd528758d2e232ab02c535ffb0: Status 404 returned error can't find the container with id 1bd242d80ee09c9e02015b09d20ace04304335cd528758d2e232ab02c535ffb0 Jul 10 17:02:37.779878 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:37.779838 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:37.845801 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:37.845767 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2lfw" event={"ID":"04084f37-a997-436d-9496-49d7f1ee8fcc","Type":"ContainerStarted","Data":"1bd242d80ee09c9e02015b09d20ace04304335cd528758d2e232ab02c535ffb0"} Jul 10 17:02:37.879466 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:37.879430 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:38.536188 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:38.536108 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:38.724007 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:38.723971 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:39.860267 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:39.860233 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2lfw" event={"ID":"04084f37-a997-436d-9496-49d7f1ee8fcc","Type":"ContainerStarted","Data":"7a22bddf1df97d7034b14743ef9a1ee2a55b59bacb93388b2a26b2fb092b73b9"} Jul 10 17:02:39.860679 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:39.860295 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2lfw" Jul 10 17:02:39.882471 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:39.882418 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2lfw" podStartSLOduration=1.977881998 podStartE2EDuration="3.882400362s" totalImagesPullingTime="1.904518364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:02:39.880161145 +0000 UTC m=+500.005869558" watchObservedRunningTime="2026-07-10 17:02:39.882400362 +0000 UTC m=+500.008108783" Jul 10 17:02:40.199953 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:40.199868 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:40.300469 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:40.300428 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:45.745707 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:45.745673 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:45.814421 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:45.814383 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:50.867517 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:50.867458 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2lfw" Jul 10 17:02:52.627179 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:52.627143 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:52.730661 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:52.730622 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:57.566290 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:57.566245 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:57.657171 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:57.657129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:59.877414 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:59.877364 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:02:59.977929 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:02:59.977895 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:05.637786 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:05.637751 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:05.701657 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:05.701619 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:13.635980 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:13.635885 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:13.743483 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:13.743436 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:18.246221 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:18.246177 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:18.341487 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:18.341452 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:18.415800 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:18.415760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:18.514623 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:18.514544 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:21.236748 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:21.236715 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:21.296224 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:21.296191 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:23.091117 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.091080 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:23.192770 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.192721 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:23.263790 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.263758 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:23.629218 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.629178 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:03:23.632165 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.632148 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-wkk95" Jul 10 17:03:23.634759 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.634725 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 17:03:23.634878 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.634826 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9g9z6\"" Jul 10 17:03:23.641118 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.641091 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:03:23.667262 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.667227 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:03:23.711383 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.711344 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d1e2a34-90a9-4d78-b0c6-fb875fdb588a-config-file\") pod \"limitador-limitador-54d6d988d5-wkk95\" (UID: \"2d1e2a34-90a9-4d78-b0c6-fb875fdb588a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wkk95" Jul 10 17:03:23.711583 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.711404 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mhx2\" (UniqueName: \"kubernetes.io/projected/2d1e2a34-90a9-4d78-b0c6-fb875fdb588a-kube-api-access-2mhx2\") pod \"limitador-limitador-54d6d988d5-wkk95\" (UID: \"2d1e2a34-90a9-4d78-b0c6-fb875fdb588a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wkk95" Jul 10 17:03:23.812742 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.812704 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mhx2\" (UniqueName: \"kubernetes.io/projected/2d1e2a34-90a9-4d78-b0c6-fb875fdb588a-kube-api-access-2mhx2\") pod \"limitador-limitador-54d6d988d5-wkk95\" (UID: \"2d1e2a34-90a9-4d78-b0c6-fb875fdb588a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wkk95" Jul 10 17:03:23.812921 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.812794 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d1e2a34-90a9-4d78-b0c6-fb875fdb588a-config-file\") pod \"limitador-limitador-54d6d988d5-wkk95\" (UID: \"2d1e2a34-90a9-4d78-b0c6-fb875fdb588a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wkk95" Jul 10 17:03:23.813394 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.813375 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d1e2a34-90a9-4d78-b0c6-fb875fdb588a-config-file\") pod \"limitador-limitador-54d6d988d5-wkk95\" (UID: \"2d1e2a34-90a9-4d78-b0c6-fb875fdb588a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wkk95" Jul 10 17:03:23.822009 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.821985 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mhx2\" (UniqueName: \"kubernetes.io/projected/2d1e2a34-90a9-4d78-b0c6-fb875fdb588a-kube-api-access-2mhx2\") pod \"limitador-limitador-54d6d988d5-wkk95\" (UID: \"2d1e2a34-90a9-4d78-b0c6-fb875fdb588a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wkk95" Jul 10 17:03:23.943316 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.943227 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-wkk95" Jul 10 17:03:23.954761 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:23.954733 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:24.029186 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:24.029148 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:24.087168 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:24.087140 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:03:24.088072 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:03:24.088042 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1e2a34_90a9_4d78_b0c6_fb875fdb588a.slice/crio-e424697f9558354fbc74bdc7f4c7c4da66fce3129f853d851e9c925ab2a14a21 WatchSource:0}: Error finding container e424697f9558354fbc74bdc7f4c7c4da66fce3129f853d851e9c925ab2a14a21: Status 404 returned error can't find the container with id e424697f9558354fbc74bdc7f4c7c4da66fce3129f853d851e9c925ab2a14a21 Jul 10 17:03:25.100572 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:25.100537 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-wkk95" event={"ID":"2d1e2a34-90a9-4d78-b0c6-fb875fdb588a","Type":"ContainerStarted","Data":"e424697f9558354fbc74bdc7f4c7c4da66fce3129f853d851e9c925ab2a14a21"} Jul 10 17:03:27.117345 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:27.117307 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-wkk95" event={"ID":"2d1e2a34-90a9-4d78-b0c6-fb875fdb588a","Type":"ContainerStarted","Data":"c7d3ba62c97e47917936a21ce8def6b5c7832c63f55636dcadb2bb80a09ff8c5"} Jul 10 17:03:27.117756 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:27.117405 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-wkk95" Jul 10 17:03:27.134934 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:27.134884 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-wkk95" podStartSLOduration=1.536321276 podStartE2EDuration="4.134869718s" totalImagesPullingTime="2.598548442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:03:27.133049216 +0000 UTC m=+547.258757627" watchObservedRunningTime="2026-07-10 17:03:27.134869718 +0000 UTC m=+547.260578128" Jul 10 17:03:27.732330 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:27.732292 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:38.124394 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:38.124362 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-wkk95" Jul 10 17:03:39.179586 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:39.179550 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:39.275302 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:39.275269 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:39.685116 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:39.685084 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:45.099684 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:45.099639 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:45.153358 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:45.153323 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:52.517658 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:52.517619 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:52.945196 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:52.945156 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:58.949020 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:58.948980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:59.052092 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:59.052059 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:59.109390 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:59.109361 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:03:59.181252 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:03:59.181211 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:04:02.568296 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:04:02.568261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:04:02.666460 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:04:02.666423 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:04:26.387029 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:04:26.386985 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:04:28.270989 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:04:28.270948 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:04:34.166317 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:04:34.166244 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:04:34.663328 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:04:34.663287 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:04:34.853242 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:04:34.853203 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:04:35.129433 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:04:35.129394 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:04:35.212669 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:04:35.212628 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:04:35.374770 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:04:35.374723 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:04:45.409869 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:04:45.409797 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:04:45.523181 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:04:45.523140 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:05:01.297835 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:01.297789 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:05:01.837431 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:01.837391 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:05:08.849786 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:08.849752 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:05:09.544977 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:09.544940 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:05:13.739628 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.739595 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl"] Jul 10 17:05:13.745672 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.743536 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.746550 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.746521 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9gt4g\"" Jul 10 17:05:13.746734 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.746578 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 17:05:13.747385 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.747359 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 10 17:05:13.747673 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.747654 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 17:05:13.754397 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.754371 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl"] Jul 10 17:05:13.768128 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.768097 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:05:13.880579 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.880542 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69lzr\" (UniqueName: \"kubernetes.io/projected/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-kube-api-access-69lzr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.880787 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.880599 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.880787 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.880633 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.880787 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.880720 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.880787 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.880783 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.880974 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.880814 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.981962 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.981927 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.981962 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.981977 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.982201 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.982026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.982201 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.982063 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.982201 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.982089 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.982201 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.982143 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69lzr\" (UniqueName: \"kubernetes.io/projected/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-kube-api-access-69lzr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.982428 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.982402 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.982514 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.982463 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.982608 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.982583 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.984659 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.984629 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.984902 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.984884 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:13.990523 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:13.990443 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69lzr\" (UniqueName: \"kubernetes.io/projected/9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b-kube-api-access-69lzr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlbsl\" (UID: \"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:14.056599 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:14.056553 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:14.192524 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:14.192483 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl"] Jul 10 17:05:14.194116 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:05:14.194088 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea9d9fa_c4c6_4986_b860_fd1aefcbac4b.slice/crio-dbb4c0656efee22dbb47ab61c2a812f0dc9f04b69e6bcdafa878d1004faecc89 WatchSource:0}: Error finding container dbb4c0656efee22dbb47ab61c2a812f0dc9f04b69e6bcdafa878d1004faecc89: Status 404 returned error can't find the container with id dbb4c0656efee22dbb47ab61c2a812f0dc9f04b69e6bcdafa878d1004faecc89 Jul 10 17:05:14.195291 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:14.195273 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:05:14.687415 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:14.687378 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" event={"ID":"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b","Type":"ContainerStarted","Data":"dbb4c0656efee22dbb47ab61c2a812f0dc9f04b69e6bcdafa878d1004faecc89"} Jul 10 17:05:14.936638 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:14.936605 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:05:15.695558 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:15.695527 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" event={"ID":"9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b","Type":"ContainerStarted","Data":"f54a16f867ade90b149202af99e69e86fa0b07980179145f2efd1b64ab26a937"} Jul 10 17:05:15.714794 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:15.714745 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" podStartSLOduration=1.598966841 podStartE2EDuration="2.714726691s" totalImagesPullingTime="1.11575985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:05:15.712267897 +0000 UTC m=+655.837976320" watchObservedRunningTime="2026-07-10 17:05:15.714726691 +0000 UTC m=+655.840435151" Jul 10 17:05:24.056935 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:24.056897 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:24.057330 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:24.056986 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:24.069553 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:24.069527 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:24.753667 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:24.753639 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlbsl" Jul 10 17:05:30.269459 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:30.269433 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:05:31.938577 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:31.938536 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:05:37.664311 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:37.664260 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:05:49.749509 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.749464 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7"] Jul 10 17:05:49.753052 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.753031 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.755721 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.755699 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 17:05:49.765274 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.765249 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7"] Jul 10 17:05:49.781167 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.781140 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:05:49.808482 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.808444 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/942b9d59-d367-488a-98df-07ebb288bfa7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.808482 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.808483 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/942b9d59-d367-488a-98df-07ebb288bfa7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.808717 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.808549 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvv7l\" (UniqueName: \"kubernetes.io/projected/942b9d59-d367-488a-98df-07ebb288bfa7-kube-api-access-nvv7l\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.808717 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.808594 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/942b9d59-d367-488a-98df-07ebb288bfa7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.808717 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.808652 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942b9d59-d367-488a-98df-07ebb288bfa7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.808717 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.808681 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/942b9d59-d367-488a-98df-07ebb288bfa7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.909768 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.909736 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/942b9d59-d367-488a-98df-07ebb288bfa7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.909965 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.909777 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/942b9d59-d367-488a-98df-07ebb288bfa7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.909965 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.909813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvv7l\" (UniqueName: \"kubernetes.io/projected/942b9d59-d367-488a-98df-07ebb288bfa7-kube-api-access-nvv7l\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.909965 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.909855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/942b9d59-d367-488a-98df-07ebb288bfa7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.909965 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.909921 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942b9d59-d367-488a-98df-07ebb288bfa7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.909965 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.909956 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/942b9d59-d367-488a-98df-07ebb288bfa7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.910358 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.910328 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942b9d59-d367-488a-98df-07ebb288bfa7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.910358 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.910366 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/942b9d59-d367-488a-98df-07ebb288bfa7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.910588 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.910413 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/942b9d59-d367-488a-98df-07ebb288bfa7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.912308 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.912283 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/942b9d59-d367-488a-98df-07ebb288bfa7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.912557 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.912538 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/942b9d59-d367-488a-98df-07ebb288bfa7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:49.921383 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:49.921353 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvv7l\" (UniqueName: \"kubernetes.io/projected/942b9d59-d367-488a-98df-07ebb288bfa7-kube-api-access-nvv7l\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9hd7\" (UID: \"942b9d59-d367-488a-98df-07ebb288bfa7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:50.063413 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:50.063325 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:05:50.205802 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:50.205770 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7"] Jul 10 17:05:50.206887 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:05:50.206865 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942b9d59_d367_488a_98df_07ebb288bfa7.slice/crio-7f72fc72d166e2913daf9f7df2148dd16aa40daf376b7978915f1a155763bc74 WatchSource:0}: Error finding container 7f72fc72d166e2913daf9f7df2148dd16aa40daf376b7978915f1a155763bc74: Status 404 returned error can't find the container with id 7f72fc72d166e2913daf9f7df2148dd16aa40daf376b7978915f1a155763bc74 Jul 10 17:05:50.883466 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:50.883428 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" event={"ID":"942b9d59-d367-488a-98df-07ebb288bfa7","Type":"ContainerStarted","Data":"76cc430d9c3c8cdaaf50649a44ac59b3cb625defeec5dd22e7556b14aa579780"} Jul 10 17:05:50.883904 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:50.883516 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" event={"ID":"942b9d59-d367-488a-98df-07ebb288bfa7","Type":"ContainerStarted","Data":"7f72fc72d166e2913daf9f7df2148dd16aa40daf376b7978915f1a155763bc74"} Jul 10 17:05:50.903668 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:50.903613 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" podStartSLOduration=1.666450294 podStartE2EDuration="1.90359625s" totalImagesPullingTime="237.145956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:05:50.901640814 +0000 UTC m=+691.027349226" watchObservedRunningTime="2026-07-10 17:05:50.90359625 +0000 UTC m=+691.029304660" Jul 10 17:05:52.038920 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:52.038886 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:05:58.952606 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:05:58.952539 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:06:00.063690 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:06:00.063653 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:06:00.064079 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:06:00.063744 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:06:00.076347 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:06:00.076322 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:06:00.944782 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:06:00.944753 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9hd7" Jul 10 17:06:10.893655 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:06:10.893562 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:06:13.335269 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:06:13.335222 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:06:33.865031 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:06:33.864996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:06:50.021087 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:06:50.021055 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:06:50.123007 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:06:50.122972 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:06:58.663808 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:06:58.663755 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:06:58.757711 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:06:58.757653 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:06:59.900478 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:06:59.900435 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:06:59.992035 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:06:59.991998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:07:27.743161 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:07:27.743110 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:07:35.958816 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:07:35.958763 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:07:37.129432 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:07:37.129397 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:07:50.227356 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:07:50.227265 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:07:58.140031 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:07:58.140002 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:08:08.236239 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:08:08.236203 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:08:10.579357 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:08:10.579323 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:08:12.156778 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:08:12.156740 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:08:35.931920 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:08:35.931884 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:08:47.632351 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:08:47.632308 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:08:53.155262 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:08:53.155223 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:08:53.252235 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:08:53.252200 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:08:53.384744 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:08:53.384710 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:08:59.367412 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:08:59.367370 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:08:59.474078 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:08:59.474029 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:08:59.573565 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:08:59.573526 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:08:59.777317 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:08:59.777192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:09:05.608185 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:09:05.608158 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:09:05.728068 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:09:05.728036 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:09:22.539695 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:09:22.539653 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:09:32.231153 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:09:32.231101 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:09:42.128242 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:09:42.128209 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:09:52.038970 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:09:52.038931 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:10:03.949652 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:10:03.949613 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:10:13.915407 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:10:13.915353 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:10:15.238429 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:10:15.238388 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:10:15.462597 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:10:15.462558 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:10:26.628645 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:10:26.628606 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:10:36.725721 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:10:36.725682 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:10:58.834089 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:10:58.834016 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:11:08.628233 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:11:08.628193 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:11:31.942944 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:11:31.942903 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:11:43.234455 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:11:43.234415 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:11:56.939153 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:11:56.939120 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:12:08.432028 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:12:08.431989 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:12:19.035416 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:12:19.035327 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:12:31.139992 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:12:31.139951 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:13:04.346049 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:13:04.346011 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:13:21.414461 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:13:21.414427 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:21.501298 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:13:21.501263 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:14:14.005192 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:14:14.005154 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:14:31.234911 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:14:31.234871 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:14:47.640571 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:14:47.640532 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:14:57.132483 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:14:57.132449 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:15:07.238311 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:15:07.238275 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:15:15.438628 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:15:15.438545 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:15:26.934830 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:15:26.934796 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:15:38.340775 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:15:38.340740 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:15:46.648141 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:15:46.648103 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:15:52.871783 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:15:52.871742 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:58.336768 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:15:58.336732 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:16:09.337913 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:16:09.337875 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:16:16.932368 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:16:16.932331 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:16:17.348325 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:16:17.348267 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:37.658006 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:16:37.657968 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:16:45.331768 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:16:45.331679 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:16:54.939970 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:16:54.939928 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:17:03.560930 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:17:03.560892 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:17:14.841592 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:17:14.841561 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:17:32.639346 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:17:32.639309 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:17:42.941618 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:17:42.941572 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:17:52.135750 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:17:52.135712 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:18:02.842800 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:18:02.842760 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:18:12.549347 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:18:12.549256 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:18:22.132856 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:18:22.132813 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:18:35.860607 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:18:35.860567 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:18:49.439205 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:18:49.439173 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:18:59.436019 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:18:59.435979 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:19:08.749128 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:19:08.749090 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:19:20.801615 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:19:20.801574 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:19:38.742373 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:19:38.742328 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:19:48.635984 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:19:48.635947 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:51.142231 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:19:51.142192 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:20:00.836431 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:00.836398 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:20:09.226844 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:09.226797 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:20:11.311287 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:11.311252 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:18.980754 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:18.980716 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:23.039472 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:23.039443 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:20:27.276052 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:27.276012 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:32.132466 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:32.132424 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:20:32.489282 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:32.489196 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:35.985070 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:35.985033 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:36.083721 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:36.083686 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:36.513485 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:36.513416 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:41.830744 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:41.830706 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:20:42.270995 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.270954 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:42.372823 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.372785 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:42.587224 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.587197 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn"] Jul 10 17:20:42.590803 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.590781 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.594016 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.593995 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8bab674c-openshift-default-dockercfg-dx2j5\"" Jul 10 17:20:42.624840 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.624806 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn"] Jul 10 17:20:42.641129 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.641092 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:42.696388 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.696346 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496f53be-7d43-4ae9-bca4-2665132df07b-istio-podinfo\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.696572 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.696397 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496f53be-7d43-4ae9-bca4-2665132df07b-istio-token\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.696572 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.696416 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-istio-data\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.696572 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.696439 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-workload-certs\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.696572 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.696484 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-workload-socket\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.696712 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.696569 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496f53be-7d43-4ae9-bca4-2665132df07b-istiod-ca-cert\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.696712 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.696647 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvz76\" (UniqueName: \"kubernetes.io/projected/496f53be-7d43-4ae9-bca4-2665132df07b-kube-api-access-hvz76\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.696712 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.696674 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-istio-envoy\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.696805 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.696744 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-credential-socket\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.740023 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.739985 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:42.798241 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.798196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496f53be-7d43-4ae9-bca4-2665132df07b-istio-podinfo\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.798438 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.798251 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496f53be-7d43-4ae9-bca4-2665132df07b-istio-token\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.798438 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.798279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-istio-data\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.798438 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.798307 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-workload-certs\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.798438 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.798348 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-workload-socket\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.798438 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.798372 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496f53be-7d43-4ae9-bca4-2665132df07b-istiod-ca-cert\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.798438 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.798397 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvz76\" (UniqueName: \"kubernetes.io/projected/496f53be-7d43-4ae9-bca4-2665132df07b-kube-api-access-hvz76\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.798791 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.798552 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-istio-envoy\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.798791 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.798641 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-credential-socket\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.798900 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.798834 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-istio-data\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.798900 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.798850 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-workload-certs\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.799051 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.798911 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-workload-socket\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.799051 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.799035 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-credential-socket\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.799556 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.799532 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496f53be-7d43-4ae9-bca4-2665132df07b-istiod-ca-cert\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.800894 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.800878 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-istio-envoy\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.800969 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.800952 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496f53be-7d43-4ae9-bca4-2665132df07b-istio-podinfo\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.806833 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.806807 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvz76\" (UniqueName: \"kubernetes.io/projected/496f53be-7d43-4ae9-bca4-2665132df07b-kube-api-access-hvz76\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.806833 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.806827 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496f53be-7d43-4ae9-bca4-2665132df07b-istio-token\") pod \"e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:42.901907 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:42.901869 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:43.045532 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:43.045474 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn"] Jul 10 17:20:43.046978 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:20:43.046947 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496f53be_7d43_4ae9_bca4_2665132df07b.slice/crio-8741c60548f44f900f922a98d75427f609dcfa782daa62426438b252c0f13e1e WatchSource:0}: Error finding container 8741c60548f44f900f922a98d75427f609dcfa782daa62426438b252c0f13e1e: Status 404 returned error can't find the container with id 8741c60548f44f900f922a98d75427f609dcfa782daa62426438b252c0f13e1e Jul 10 17:20:43.049227 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:43.049186 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:20:43.049316 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:43.049277 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:20:43.049354 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:43.049320 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:20:43.637408 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:43.637369 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" event={"ID":"496f53be-7d43-4ae9-bca4-2665132df07b","Type":"ContainerStarted","Data":"8e16d81da276de4cd53747f592e1620de419b4cfa66a1a68f967c9b251e95c23"} Jul 10 17:20:43.637616 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:43.637445 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" event={"ID":"496f53be-7d43-4ae9-bca4-2665132df07b","Type":"ContainerStarted","Data":"8741c60548f44f900f922a98d75427f609dcfa782daa62426438b252c0f13e1e"} Jul 10 17:20:43.659350 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:43.659291 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" podStartSLOduration=1.659275474 podStartE2EDuration="1.659275474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:20:43.656913324 +0000 UTC m=+1583.782621735" watchObservedRunningTime="2026-07-10 17:20:43.659275474 +0000 UTC m=+1583.784983875" Jul 10 17:20:43.738995 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:43.738948 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:43.839356 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:43.839312 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:43.903081 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:43.902991 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:43.960488 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:43.960453 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:44.183843 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:44.183763 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:44.380877 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:44.380838 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:44.484813 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:44.484733 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:44.907750 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:44.907718 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:45.649466 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:45.649434 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:45.650532 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:45.650485 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:46.148151 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:46.148113 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:48.946433 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:48.946398 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:49.142869 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:49.142834 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn"] Jul 10 17:20:49.143105 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:49.143075 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" podUID="496f53be-7d43-4ae9-bca4-2665132df07b" containerName="istio-proxy" containerID="cri-o://8e16d81da276de4cd53747f592e1620de419b4cfa66a1a68f967c9b251e95c23" gracePeriod=30 Jul 10 17:20:49.171529 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:49.171471 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:49.302440 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:49.302345 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:49.356801 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:49.356764 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:50.674373 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:50.674322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:55.027312 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.027274 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:55.297150 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.297125 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:55.303170 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303147 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-credential-socket\") pod \"496f53be-7d43-4ae9-bca4-2665132df07b\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " Jul 10 17:20:55.303312 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303176 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-workload-socket\") pod \"496f53be-7d43-4ae9-bca4-2665132df07b\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " Jul 10 17:20:55.303312 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303211 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496f53be-7d43-4ae9-bca4-2665132df07b-istio-podinfo\") pod \"496f53be-7d43-4ae9-bca4-2665132df07b\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " Jul 10 17:20:55.303312 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303262 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-workload-certs\") pod \"496f53be-7d43-4ae9-bca4-2665132df07b\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " Jul 10 17:20:55.303312 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303291 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvz76\" (UniqueName: \"kubernetes.io/projected/496f53be-7d43-4ae9-bca4-2665132df07b-kube-api-access-hvz76\") pod \"496f53be-7d43-4ae9-bca4-2665132df07b\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " Jul 10 17:20:55.303554 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303325 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496f53be-7d43-4ae9-bca4-2665132df07b-istio-token\") pod \"496f53be-7d43-4ae9-bca4-2665132df07b\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " Jul 10 17:20:55.303554 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303368 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-istio-envoy\") pod \"496f53be-7d43-4ae9-bca4-2665132df07b\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " Jul 10 17:20:55.303554 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303392 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496f53be-7d43-4ae9-bca4-2665132df07b-istiod-ca-cert\") pod \"496f53be-7d43-4ae9-bca4-2665132df07b\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " Jul 10 17:20:55.303554 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303433 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-istio-data\") pod \"496f53be-7d43-4ae9-bca4-2665132df07b\" (UID: \"496f53be-7d43-4ae9-bca4-2665132df07b\") " Jul 10 17:20:55.303554 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303456 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "496f53be-7d43-4ae9-bca4-2665132df07b" (UID: "496f53be-7d43-4ae9-bca4-2665132df07b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:20:55.303554 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303468 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "496f53be-7d43-4ae9-bca4-2665132df07b" (UID: "496f53be-7d43-4ae9-bca4-2665132df07b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:20:55.303872 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303600 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "496f53be-7d43-4ae9-bca4-2665132df07b" (UID: "496f53be-7d43-4ae9-bca4-2665132df07b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:20:55.303872 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303814 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-workload-certs\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:20:55.303872 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303838 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-credential-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:20:55.303872 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303854 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-workload-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:20:55.304248 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.303906 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496f53be-7d43-4ae9-bca4-2665132df07b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "496f53be-7d43-4ae9-bca4-2665132df07b" (UID: "496f53be-7d43-4ae9-bca4-2665132df07b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:20:55.304594 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.304565 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-istio-data" (OuterVolumeSpecName: "istio-data") pod "496f53be-7d43-4ae9-bca4-2665132df07b" (UID: "496f53be-7d43-4ae9-bca4-2665132df07b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:20:55.306002 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.305967 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "496f53be-7d43-4ae9-bca4-2665132df07b" (UID: "496f53be-7d43-4ae9-bca4-2665132df07b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:20:55.306090 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.306011 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496f53be-7d43-4ae9-bca4-2665132df07b-kube-api-access-hvz76" (OuterVolumeSpecName: "kube-api-access-hvz76") pod "496f53be-7d43-4ae9-bca4-2665132df07b" (UID: "496f53be-7d43-4ae9-bca4-2665132df07b"). InnerVolumeSpecName "kube-api-access-hvz76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:20:55.306090 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.306059 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496f53be-7d43-4ae9-bca4-2665132df07b-istio-token" (OuterVolumeSpecName: "istio-token") pod "496f53be-7d43-4ae9-bca4-2665132df07b" (UID: "496f53be-7d43-4ae9-bca4-2665132df07b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:20:55.306090 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.306067 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/496f53be-7d43-4ae9-bca4-2665132df07b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "496f53be-7d43-4ae9-bca4-2665132df07b" (UID: "496f53be-7d43-4ae9-bca4-2665132df07b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:20:55.404509 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.404374 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496f53be-7d43-4ae9-bca4-2665132df07b-istio-podinfo\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:20:55.404509 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.404410 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvz76\" (UniqueName: \"kubernetes.io/projected/496f53be-7d43-4ae9-bca4-2665132df07b-kube-api-access-hvz76\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:20:55.404509 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.404427 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496f53be-7d43-4ae9-bca4-2665132df07b-istio-token\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:20:55.404509 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.404443 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-istio-envoy\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:20:55.404509 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.404455 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496f53be-7d43-4ae9-bca4-2665132df07b-istiod-ca-cert\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:20:55.404509 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.404466 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496f53be-7d43-4ae9-bca4-2665132df07b-istio-data\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:20:55.714991 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.714953 2559 generic.go:356] "Generic (PLEG): container finished" podID="496f53be-7d43-4ae9-bca4-2665132df07b" containerID="8e16d81da276de4cd53747f592e1620de419b4cfa66a1a68f967c9b251e95c23" exitCode=0 Jul 10 17:20:55.715164 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.715005 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" Jul 10 17:20:55.715164 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.715045 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" event={"ID":"496f53be-7d43-4ae9-bca4-2665132df07b","Type":"ContainerDied","Data":"8e16d81da276de4cd53747f592e1620de419b4cfa66a1a68f967c9b251e95c23"} Jul 10 17:20:55.715164 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.715111 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn" event={"ID":"496f53be-7d43-4ae9-bca4-2665132df07b","Type":"ContainerDied","Data":"8741c60548f44f900f922a98d75427f609dcfa782daa62426438b252c0f13e1e"} Jul 10 17:20:55.715164 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.715134 2559 scope.go:117] "RemoveContainer" containerID="8e16d81da276de4cd53747f592e1620de419b4cfa66a1a68f967c9b251e95c23" Jul 10 17:20:55.725177 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.725157 2559 scope.go:117] "RemoveContainer" containerID="8e16d81da276de4cd53747f592e1620de419b4cfa66a1a68f967c9b251e95c23" Jul 10 17:20:55.725460 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:20:55.725437 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e16d81da276de4cd53747f592e1620de419b4cfa66a1a68f967c9b251e95c23\": container with ID starting with 8e16d81da276de4cd53747f592e1620de419b4cfa66a1a68f967c9b251e95c23 not found: ID does not exist" containerID="8e16d81da276de4cd53747f592e1620de419b4cfa66a1a68f967c9b251e95c23" Jul 10 17:20:55.725591 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.725468 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e16d81da276de4cd53747f592e1620de419b4cfa66a1a68f967c9b251e95c23"} err="failed to get container status \"8e16d81da276de4cd53747f592e1620de419b4cfa66a1a68f967c9b251e95c23\": rpc error: code = NotFound desc = could not find container \"8e16d81da276de4cd53747f592e1620de419b4cfa66a1a68f967c9b251e95c23\": container with ID starting with 8e16d81da276de4cd53747f592e1620de419b4cfa66a1a68f967c9b251e95c23 not found: ID does not exist" Jul 10 17:20:55.739517 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.739476 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn"] Jul 10 17:20:55.743338 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:55.743314 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8bab674c-openshift-default-5579d598df-4pmjn"] Jul 10 17:20:56.398766 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:56.398732 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="496f53be-7d43-4ae9-bca4-2665132df07b" path="/var/lib/kubelet/pods/496f53be-7d43-4ae9-bca4-2665132df07b/volumes" Jul 10 17:20:56.540700 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:56.540667 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:56.772901 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:56.772806 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:56.875678 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:56.875626 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:56.978394 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:56.978355 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:57.346562 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:57.346528 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:57.447104 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:57.447069 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:58.801723 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:58.801691 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:59.731326 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:20:59.731288 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:21:01.474759 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:01.474723 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:01.706560 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:01.706528 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:01.838379 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:01.838351 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:01.939677 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:01.939643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:03.432277 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:03.432237 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:07.499030 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.498990 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:07.598833 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.598795 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:07.705158 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.705127 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9"] Jul 10 17:21:07.705624 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.705605 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496f53be-7d43-4ae9-bca4-2665132df07b" containerName="istio-proxy" Jul 10 17:21:07.705624 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.705623 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="496f53be-7d43-4ae9-bca4-2665132df07b" containerName="istio-proxy" Jul 10 17:21:07.705813 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.705716 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="496f53be-7d43-4ae9-bca4-2665132df07b" containerName="istio-proxy" Jul 10 17:21:07.708862 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.708839 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.711438 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.711412 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-45063052-openshift-default-dockercfg-2bvv8\"" Jul 10 17:21:07.721213 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.721188 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9"] Jul 10 17:21:07.747915 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.747880 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:07.807930 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.807827 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-workload-socket\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.807930 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.807871 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-credential-socket\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.807930 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.807896 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-workload-certs\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.807930 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.807917 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-podinfo\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.808232 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.807944 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-envoy\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.808232 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.808049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67b2f\" (UniqueName: \"kubernetes.io/projected/37db5a27-984f-4473-86b2-b2c02a5530d5-kube-api-access-67b2f\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.808232 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.808106 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-token\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.808232 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.808136 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37db5a27-984f-4473-86b2-b2c02a5530d5-istiod-ca-cert\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.808232 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.808159 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-data\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.814658 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.814625 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:07.909621 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.909585 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67b2f\" (UniqueName: \"kubernetes.io/projected/37db5a27-984f-4473-86b2-b2c02a5530d5-kube-api-access-67b2f\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.909833 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.909640 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-token\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.909833 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.909662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37db5a27-984f-4473-86b2-b2c02a5530d5-istiod-ca-cert\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.909833 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.909682 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-data\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.909833 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.909717 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-workload-socket\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.909833 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.909732 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-credential-socket\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.909833 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.909761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-workload-certs\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.910159 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.909902 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-podinfo\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.910159 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.909956 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-envoy\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.910159 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.910131 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-workload-certs\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.910387 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.910165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-workload-socket\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.910387 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.910219 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-credential-socket\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.910387 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.910356 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-data\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.910614 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.910591 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37db5a27-984f-4473-86b2-b2c02a5530d5-istiod-ca-cert\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.912382 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.912361 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-envoy\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.912625 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.912605 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-podinfo\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.921439 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.921413 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-token\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:07.921583 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:07.921458 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67b2f\" (UniqueName: \"kubernetes.io/projected/37db5a27-984f-4473-86b2-b2c02a5530d5-kube-api-access-67b2f\") pod \"e2e-ait-45063052-openshift-default-79496b5457-6zxr9\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:08.021296 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:08.021261 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:08.162479 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:08.162451 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9"] Jul 10 17:21:08.164175 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:21:08.164148 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37db5a27_984f_4473_86b2_b2c02a5530d5.slice/crio-b0e3d87b5268be0d7a5a7b30c4f5983559e83888bfd8284d61c04e5e116aea03 WatchSource:0}: Error finding container b0e3d87b5268be0d7a5a7b30c4f5983559e83888bfd8284d61c04e5e116aea03: Status 404 returned error can't find the container with id b0e3d87b5268be0d7a5a7b30c4f5983559e83888bfd8284d61c04e5e116aea03 Jul 10 17:21:08.166310 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:08.166277 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:21:08.166390 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:08.166344 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:21:08.166390 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:08.166370 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:21:08.785996 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:08.785956 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" event={"ID":"37db5a27-984f-4473-86b2-b2c02a5530d5","Type":"ContainerStarted","Data":"2964b1d4794906b4ce8e10e4db4dd87fc72d2db4aa349f50d822ed1875208649"} Jul 10 17:21:08.786443 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:08.786024 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" event={"ID":"37db5a27-984f-4473-86b2-b2c02a5530d5","Type":"ContainerStarted","Data":"b0e3d87b5268be0d7a5a7b30c4f5983559e83888bfd8284d61c04e5e116aea03"} Jul 10 17:21:08.809349 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:08.809299 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" podStartSLOduration=1.809282911 podStartE2EDuration="1.809282911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:21:08.807352522 +0000 UTC m=+1608.933060944" watchObservedRunningTime="2026-07-10 17:21:08.809282911 +0000 UTC m=+1608.934991331" Jul 10 17:21:08.857662 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:08.857627 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:08.946065 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:08.946001 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:09.010452 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:09.010416 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:09.022082 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:09.022056 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:09.027111 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:09.027083 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:09.111150 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:09.111116 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:09.598544 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:09.598513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:09.696976 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:09.696937 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:09.791524 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:09.791469 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:09.792433 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:09.792418 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:10.938627 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:10.938594 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:13.031789 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:13.031744 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:21:14.206248 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:14.206212 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:14.564354 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:14.564224 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:20.437551 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:20.437483 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9"] Jul 10 17:21:20.438039 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:20.437740 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" podUID="37db5a27-984f-4473-86b2-b2c02a5530d5" containerName="istio-proxy" containerID="cri-o://2964b1d4794906b4ce8e10e4db4dd87fc72d2db4aa349f50d822ed1875208649" gracePeriod=30 Jul 10 17:21:20.461828 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:20.461794 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:20.571266 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:20.571226 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:21.483313 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.483283 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:21.547251 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.547216 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-podinfo\") pod \"37db5a27-984f-4473-86b2-b2c02a5530d5\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " Jul 10 17:21:21.547445 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.547279 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-credential-socket\") pod \"37db5a27-984f-4473-86b2-b2c02a5530d5\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " Jul 10 17:21:21.547445 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.547301 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-envoy\") pod \"37db5a27-984f-4473-86b2-b2c02a5530d5\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " Jul 10 17:21:21.547445 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.547337 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-workload-certs\") pod \"37db5a27-984f-4473-86b2-b2c02a5530d5\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " Jul 10 17:21:21.547445 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.547360 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-data\") pod \"37db5a27-984f-4473-86b2-b2c02a5530d5\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " Jul 10 17:21:21.547445 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.547388 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67b2f\" (UniqueName: \"kubernetes.io/projected/37db5a27-984f-4473-86b2-b2c02a5530d5-kube-api-access-67b2f\") pod \"37db5a27-984f-4473-86b2-b2c02a5530d5\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " Jul 10 17:21:21.547445 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.547444 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-workload-socket\") pod \"37db5a27-984f-4473-86b2-b2c02a5530d5\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " Jul 10 17:21:21.547796 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.547554 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-token\") pod \"37db5a27-984f-4473-86b2-b2c02a5530d5\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " Jul 10 17:21:21.547796 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.547579 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37db5a27-984f-4473-86b2-b2c02a5530d5-istiod-ca-cert\") pod \"37db5a27-984f-4473-86b2-b2c02a5530d5\" (UID: \"37db5a27-984f-4473-86b2-b2c02a5530d5\") " Jul 10 17:21:21.547796 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.547684 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "37db5a27-984f-4473-86b2-b2c02a5530d5" (UID: "37db5a27-984f-4473-86b2-b2c02a5530d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:21.547796 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.547697 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "37db5a27-984f-4473-86b2-b2c02a5530d5" (UID: "37db5a27-984f-4473-86b2-b2c02a5530d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:21.548346 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.547971 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-credential-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:21.548346 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.547992 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-workload-certs\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:21.548346 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.548062 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37db5a27-984f-4473-86b2-b2c02a5530d5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "37db5a27-984f-4473-86b2-b2c02a5530d5" (UID: "37db5a27-984f-4473-86b2-b2c02a5530d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:21:21.548346 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.548073 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "37db5a27-984f-4473-86b2-b2c02a5530d5" (UID: "37db5a27-984f-4473-86b2-b2c02a5530d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:21.548738 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.548697 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-data" (OuterVolumeSpecName: "istio-data") pod "37db5a27-984f-4473-86b2-b2c02a5530d5" (UID: "37db5a27-984f-4473-86b2-b2c02a5530d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:21.550208 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.550163 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "37db5a27-984f-4473-86b2-b2c02a5530d5" (UID: "37db5a27-984f-4473-86b2-b2c02a5530d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:21:21.550208 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.550175 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37db5a27-984f-4473-86b2-b2c02a5530d5-kube-api-access-67b2f" (OuterVolumeSpecName: "kube-api-access-67b2f") pod "37db5a27-984f-4473-86b2-b2c02a5530d5" (UID: "37db5a27-984f-4473-86b2-b2c02a5530d5"). InnerVolumeSpecName "kube-api-access-67b2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:21:21.550629 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.550606 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "37db5a27-984f-4473-86b2-b2c02a5530d5" (UID: "37db5a27-984f-4473-86b2-b2c02a5530d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:21.550730 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.550631 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "37db5a27-984f-4473-86b2-b2c02a5530d5" (UID: "37db5a27-984f-4473-86b2-b2c02a5530d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:21:21.648788 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.648747 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-workload-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:21.648788 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.648788 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-token\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:21.648982 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.648802 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37db5a27-984f-4473-86b2-b2c02a5530d5-istiod-ca-cert\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:21.648982 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.648814 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-podinfo\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:21.648982 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.648826 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-envoy\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:21.648982 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.648838 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37db5a27-984f-4473-86b2-b2c02a5530d5-istio-data\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:21.648982 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.648852 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67b2f\" (UniqueName: \"kubernetes.io/projected/37db5a27-984f-4473-86b2-b2c02a5530d5-kube-api-access-67b2f\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:21.860989 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.860955 2559 generic.go:356] "Generic (PLEG): container finished" podID="37db5a27-984f-4473-86b2-b2c02a5530d5" containerID="2964b1d4794906b4ce8e10e4db4dd87fc72d2db4aa349f50d822ed1875208649" exitCode=0 Jul 10 17:21:21.861174 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.861007 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" Jul 10 17:21:21.861174 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.861044 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" event={"ID":"37db5a27-984f-4473-86b2-b2c02a5530d5","Type":"ContainerDied","Data":"2964b1d4794906b4ce8e10e4db4dd87fc72d2db4aa349f50d822ed1875208649"} Jul 10 17:21:21.861174 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.861100 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9" event={"ID":"37db5a27-984f-4473-86b2-b2c02a5530d5","Type":"ContainerDied","Data":"b0e3d87b5268be0d7a5a7b30c4f5983559e83888bfd8284d61c04e5e116aea03"} Jul 10 17:21:21.861174 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.861122 2559 scope.go:117] "RemoveContainer" containerID="2964b1d4794906b4ce8e10e4db4dd87fc72d2db4aa349f50d822ed1875208649" Jul 10 17:21:21.871136 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.871115 2559 scope.go:117] "RemoveContainer" containerID="2964b1d4794906b4ce8e10e4db4dd87fc72d2db4aa349f50d822ed1875208649" Jul 10 17:21:21.871413 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:21:21.871390 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2964b1d4794906b4ce8e10e4db4dd87fc72d2db4aa349f50d822ed1875208649\": container with ID starting with 2964b1d4794906b4ce8e10e4db4dd87fc72d2db4aa349f50d822ed1875208649 not found: ID does not exist" containerID="2964b1d4794906b4ce8e10e4db4dd87fc72d2db4aa349f50d822ed1875208649" Jul 10 17:21:21.871471 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.871423 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2964b1d4794906b4ce8e10e4db4dd87fc72d2db4aa349f50d822ed1875208649"} err="failed to get container status \"2964b1d4794906b4ce8e10e4db4dd87fc72d2db4aa349f50d822ed1875208649\": rpc error: code = NotFound desc = could not find container \"2964b1d4794906b4ce8e10e4db4dd87fc72d2db4aa349f50d822ed1875208649\": container with ID starting with 2964b1d4794906b4ce8e10e4db4dd87fc72d2db4aa349f50d822ed1875208649 not found: ID does not exist" Jul 10 17:21:21.883152 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.883123 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9"] Jul 10 17:21:21.887196 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:21.887166 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-45063052-openshift-default-79496b5457-6zxr9"] Jul 10 17:21:22.400068 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:22.400034 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37db5a27-984f-4473-86b2-b2c02a5530d5" path="/var/lib/kubelet/pods/37db5a27-984f-4473-86b2-b2c02a5530d5/volumes" Jul 10 17:21:22.403915 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:22.403883 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:23.334621 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:23.334583 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:21:26.223333 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.223297 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:26.282753 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.282720 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:26.418233 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.418175 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw"] Jul 10 17:21:26.419058 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.419033 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37db5a27-984f-4473-86b2-b2c02a5530d5" containerName="istio-proxy" Jul 10 17:21:26.419214 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.419202 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="37db5a27-984f-4473-86b2-b2c02a5530d5" containerName="istio-proxy" Jul 10 17:21:26.419444 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.419431 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="37db5a27-984f-4473-86b2-b2c02a5530d5" containerName="istio-proxy" Jul 10 17:21:26.426340 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.426317 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.438841 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.438188 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw"] Jul 10 17:21:26.440604 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.440124 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-7dfb6652-openshift-default-dockercfg-6dzrx\"" Jul 10 17:21:26.448240 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.448214 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:26.490481 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.490390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-workload-socket\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.490481 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.490435 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-data\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.490481 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.490469 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-token\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.490715 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.490576 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-credential-socket\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.490715 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.490628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-podinfo\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.490803 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.490715 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71a7a88-ad81-4555-b3d2-97910d47c896-istiod-ca-cert\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.490803 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.490751 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zsf\" (UniqueName: \"kubernetes.io/projected/c71a7a88-ad81-4555-b3d2-97910d47c896-kube-api-access-z7zsf\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.490803 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.490790 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-envoy\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.490968 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.490812 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-workload-certs\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.548767 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.548730 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:26.592137 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592099 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71a7a88-ad81-4555-b3d2-97910d47c896-istiod-ca-cert\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.592137 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592141 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zsf\" (UniqueName: \"kubernetes.io/projected/c71a7a88-ad81-4555-b3d2-97910d47c896-kube-api-access-z7zsf\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.592390 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-envoy\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.592390 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-workload-certs\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.592390 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592233 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-workload-socket\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.592390 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592258 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-data\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.592390 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592311 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-token\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.592714 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-credential-socket\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.592714 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592455 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-podinfo\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.592714 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592673 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-workload-socket\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.592868 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592729 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-data\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.592868 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592819 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-credential-socket\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.592868 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592844 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-workload-certs\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.592985 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.592967 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71a7a88-ad81-4555-b3d2-97910d47c896-istiod-ca-cert\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.595000 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.594976 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-envoy\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.595197 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.595180 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-podinfo\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.600812 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.600778 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-token\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.600933 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.600901 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zsf\" (UniqueName: \"kubernetes.io/projected/c71a7a88-ad81-4555-b3d2-97910d47c896-kube-api-access-z7zsf\") pod \"e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.748247 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.748160 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:26.888834 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.888791 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw"] Jul 10 17:21:26.890233 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:21:26.890192 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71a7a88_ad81_4555_b3d2_97910d47c896.slice/crio-3d46ac2cbde8f3835ddd375035cfd049eb9a1595de3b9a07c3f00018724eec9c WatchSource:0}: Error finding container 3d46ac2cbde8f3835ddd375035cfd049eb9a1595de3b9a07c3f00018724eec9c: Status 404 returned error can't find the container with id 3d46ac2cbde8f3835ddd375035cfd049eb9a1595de3b9a07c3f00018724eec9c Jul 10 17:21:26.892251 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.892216 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:21:26.892361 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.892293 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:21:26.892361 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:26.892340 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:21:27.559331 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:27.559293 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:27.636803 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:27.636763 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:27.729460 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:27.729424 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:27.898706 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:27.898669 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" event={"ID":"c71a7a88-ad81-4555-b3d2-97910d47c896","Type":"ContainerStarted","Data":"9434f6ddf36cb5f5a854b0411b4ba3bbcabcfe41d40b78427e926beb47a36c64"} Jul 10 17:21:27.898911 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:27.898731 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" event={"ID":"c71a7a88-ad81-4555-b3d2-97910d47c896","Type":"ContainerStarted","Data":"3d46ac2cbde8f3835ddd375035cfd049eb9a1595de3b9a07c3f00018724eec9c"} Jul 10 17:21:28.185934 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:28.185850 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:28.284261 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:28.284230 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:28.384607 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:28.384581 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:28.748853 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:28.748810 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:28.753592 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:28.753564 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:28.775064 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:28.775010 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" podStartSLOduration=2.774993691 podStartE2EDuration="2.774993691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:21:27.918559546 +0000 UTC m=+1628.044267957" watchObservedRunningTime="2026-07-10 17:21:28.774993691 +0000 UTC m=+1628.900702162" Jul 10 17:21:28.904836 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:28.904811 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:28.905638 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:28.905615 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:29.433384 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:29.433351 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:31.031211 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:31.031174 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:21:32.127258 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:32.127218 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:32.378113 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:32.378026 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw"] Jul 10 17:21:32.378266 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:32.378212 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" podUID="c71a7a88-ad81-4555-b3d2-97910d47c896" containerName="istio-proxy" containerID="cri-o://9434f6ddf36cb5f5a854b0411b4ba3bbcabcfe41d40b78427e926beb47a36c64" gracePeriod=30 Jul 10 17:21:32.429808 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:32.429766 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:32.526532 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:32.526465 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:32.611174 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:32.611089 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:35.672728 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:35.672689 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:38.324184 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.324068 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:38.425289 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.424355 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:38.738764 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.738736 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:38.880514 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.880400 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:38.912003 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.911967 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-credential-socket\") pod \"c71a7a88-ad81-4555-b3d2-97910d47c896\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " Jul 10 17:21:38.912180 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.912019 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-data\") pod \"c71a7a88-ad81-4555-b3d2-97910d47c896\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " Jul 10 17:21:38.912180 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.912037 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71a7a88-ad81-4555-b3d2-97910d47c896-istiod-ca-cert\") pod \"c71a7a88-ad81-4555-b3d2-97910d47c896\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " Jul 10 17:21:38.912180 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.912072 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-podinfo\") pod \"c71a7a88-ad81-4555-b3d2-97910d47c896\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " Jul 10 17:21:38.912180 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.912115 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-envoy\") pod \"c71a7a88-ad81-4555-b3d2-97910d47c896\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " Jul 10 17:21:38.912180 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.912164 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-workload-socket\") pod \"c71a7a88-ad81-4555-b3d2-97910d47c896\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " Jul 10 17:21:38.912180 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.912181 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7zsf\" (UniqueName: \"kubernetes.io/projected/c71a7a88-ad81-4555-b3d2-97910d47c896-kube-api-access-z7zsf\") pod \"c71a7a88-ad81-4555-b3d2-97910d47c896\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " Jul 10 17:21:38.912469 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.912200 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-workload-certs\") pod \"c71a7a88-ad81-4555-b3d2-97910d47c896\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " Jul 10 17:21:38.912469 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.912233 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-token\") pod \"c71a7a88-ad81-4555-b3d2-97910d47c896\" (UID: \"c71a7a88-ad81-4555-b3d2-97910d47c896\") " Jul 10 17:21:38.912768 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.912345 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c71a7a88-ad81-4555-b3d2-97910d47c896" (UID: "c71a7a88-ad81-4555-b3d2-97910d47c896"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:38.912864 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.912518 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c71a7a88-ad81-4555-b3d2-97910d47c896" (UID: "c71a7a88-ad81-4555-b3d2-97910d47c896"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:38.912864 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.912530 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71a7a88-ad81-4555-b3d2-97910d47c896-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c71a7a88-ad81-4555-b3d2-97910d47c896" (UID: "c71a7a88-ad81-4555-b3d2-97910d47c896"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:21:38.912864 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.912594 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c71a7a88-ad81-4555-b3d2-97910d47c896" (UID: "c71a7a88-ad81-4555-b3d2-97910d47c896"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:38.913191 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.913017 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-data" (OuterVolumeSpecName: "istio-data") pod "c71a7a88-ad81-4555-b3d2-97910d47c896" (UID: "c71a7a88-ad81-4555-b3d2-97910d47c896"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:38.913256 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.913211 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-workload-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:38.913256 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.913235 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-workload-certs\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:38.913256 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.913250 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-credential-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:38.913419 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.913263 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71a7a88-ad81-4555-b3d2-97910d47c896-istiod-ca-cert\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:38.914737 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.914709 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-token" (OuterVolumeSpecName: "istio-token") pod "c71a7a88-ad81-4555-b3d2-97910d47c896" (UID: "c71a7a88-ad81-4555-b3d2-97910d47c896"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:21:38.914863 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.914745 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71a7a88-ad81-4555-b3d2-97910d47c896-kube-api-access-z7zsf" (OuterVolumeSpecName: "kube-api-access-z7zsf") pod "c71a7a88-ad81-4555-b3d2-97910d47c896" (UID: "c71a7a88-ad81-4555-b3d2-97910d47c896"). InnerVolumeSpecName "kube-api-access-z7zsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:21:38.914969 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.914953 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c71a7a88-ad81-4555-b3d2-97910d47c896" (UID: "c71a7a88-ad81-4555-b3d2-97910d47c896"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:21:38.915029 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.915015 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c71a7a88-ad81-4555-b3d2-97910d47c896" (UID: "c71a7a88-ad81-4555-b3d2-97910d47c896"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:38.963462 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.963423 2559 generic.go:356] "Generic (PLEG): container finished" podID="c71a7a88-ad81-4555-b3d2-97910d47c896" containerID="9434f6ddf36cb5f5a854b0411b4ba3bbcabcfe41d40b78427e926beb47a36c64" exitCode=0 Jul 10 17:21:38.963462 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.963468 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" Jul 10 17:21:38.963734 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.963517 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" event={"ID":"c71a7a88-ad81-4555-b3d2-97910d47c896","Type":"ContainerDied","Data":"9434f6ddf36cb5f5a854b0411b4ba3bbcabcfe41d40b78427e926beb47a36c64"} Jul 10 17:21:38.963734 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.963571 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw" event={"ID":"c71a7a88-ad81-4555-b3d2-97910d47c896","Type":"ContainerDied","Data":"3d46ac2cbde8f3835ddd375035cfd049eb9a1595de3b9a07c3f00018724eec9c"} Jul 10 17:21:38.963734 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.963596 2559 scope.go:117] "RemoveContainer" containerID="9434f6ddf36cb5f5a854b0411b4ba3bbcabcfe41d40b78427e926beb47a36c64" Jul 10 17:21:38.973245 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.973224 2559 scope.go:117] "RemoveContainer" containerID="9434f6ddf36cb5f5a854b0411b4ba3bbcabcfe41d40b78427e926beb47a36c64" Jul 10 17:21:38.973555 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:21:38.973533 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9434f6ddf36cb5f5a854b0411b4ba3bbcabcfe41d40b78427e926beb47a36c64\": container with ID starting with 9434f6ddf36cb5f5a854b0411b4ba3bbcabcfe41d40b78427e926beb47a36c64 not found: ID does not exist" containerID="9434f6ddf36cb5f5a854b0411b4ba3bbcabcfe41d40b78427e926beb47a36c64" Jul 10 17:21:38.973620 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.973564 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9434f6ddf36cb5f5a854b0411b4ba3bbcabcfe41d40b78427e926beb47a36c64"} err="failed to get container status \"9434f6ddf36cb5f5a854b0411b4ba3bbcabcfe41d40b78427e926beb47a36c64\": rpc error: code = NotFound desc = could not find container \"9434f6ddf36cb5f5a854b0411b4ba3bbcabcfe41d40b78427e926beb47a36c64\": container with ID starting with 9434f6ddf36cb5f5a854b0411b4ba3bbcabcfe41d40b78427e926beb47a36c64 not found: ID does not exist" Jul 10 17:21:38.975406 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.975375 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:38.985102 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.985079 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw"] Jul 10 17:21:38.994520 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:38.992267 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-7dfb6652-openshift-default-788bd9b7c9-gw4nw"] Jul 10 17:21:39.013767 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:39.013738 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-data\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:39.013767 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:39.013761 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-podinfo\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:39.013767 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:39.013770 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-envoy\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:39.013962 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:39.013782 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7zsf\" (UniqueName: \"kubernetes.io/projected/c71a7a88-ad81-4555-b3d2-97910d47c896-kube-api-access-z7zsf\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:39.013962 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:39.013791 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71a7a88-ad81-4555-b3d2-97910d47c896-istio-token\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:21:39.030301 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:39.030273 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:40.399314 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:40.399282 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c71a7a88-ad81-4555-b3d2-97910d47c896" path="/var/lib/kubelet/pods/c71a7a88-ad81-4555-b3d2-97910d47c896/volumes" Jul 10 17:21:40.657253 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:40.657167 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:42.333666 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:42.333631 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:21:45.736478 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:45.736433 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:45.838423 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:45.838392 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:46.819632 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:46.819595 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:46.926775 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:46.926740 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:47.022064 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:47.022029 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:47.120911 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:47.120882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:47.677474 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:47.677442 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:53.838651 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:21:53.838615 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:22:13.729211 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:13.729176 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:19.493771 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:19.493735 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:19.594746 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:19.594712 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:19.693691 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:19.693652 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:34.745011 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:34.744976 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:34.846136 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:34.846096 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:35.723514 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:35.723456 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:41.443051 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:41.442957 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:41.549462 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:41.549414 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:41.643076 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:41.643038 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:52.205670 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:52.205633 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:53.124929 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:53.124889 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:58.792002 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:58.791969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:58.892144 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:58.892109 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:58.992215 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:58.992177 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:59.096679 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:59.096641 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:59.945007 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:22:59.944973 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:00.044542 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:00.044487 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:00.143475 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:00.143435 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:01.136614 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:01.136577 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:23:01.972437 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:01.972402 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:08.092471 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:08.092435 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:08.168135 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:08.168074 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:08.682587 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:08.682548 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:08.785640 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:08.785592 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:08.885201 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:08.885168 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:09.150806 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:09.150772 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:23:14.794952 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:14.794914 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:14.889906 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:14.889867 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:15.056936 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.056833 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:15.377510 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.377464 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g"] Jul 10 17:23:15.377868 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.377853 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71a7a88-ad81-4555-b3d2-97910d47c896" containerName="istio-proxy" Jul 10 17:23:15.377868 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.377869 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71a7a88-ad81-4555-b3d2-97910d47c896" containerName="istio-proxy" Jul 10 17:23:15.377965 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.377945 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71a7a88-ad81-4555-b3d2-97910d47c896" containerName="istio-proxy" Jul 10 17:23:15.380159 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.380140 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.383033 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.383001 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3ff7ed0c-openshift-default-dockercfg-wlr5c\"" Jul 10 17:23:15.408466 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.408429 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g"] Jul 10 17:23:15.416712 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.416681 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:15.478485 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.478444 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-workload-socket\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.478687 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.478484 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-data\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.478687 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.478582 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-workload-certs\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.478687 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.478613 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-token\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.478687 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.478640 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vfwt\" (UniqueName: \"kubernetes.io/projected/05fa6cec-639f-4ac7-8237-2edefd161dc7-kube-api-access-9vfwt\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.478687 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.478672 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fa6cec-639f-4ac7-8237-2edefd161dc7-istiod-ca-cert\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.478898 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.478697 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-podinfo\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.478898 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.478744 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-envoy\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.478898 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.478763 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-credential-socket\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.580320 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.580276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fa6cec-639f-4ac7-8237-2edefd161dc7-istiod-ca-cert\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.580569 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.580332 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-podinfo\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.580569 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.580386 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-envoy\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.580569 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.580413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-credential-socket\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.580569 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.580486 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-workload-socket\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.580569 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.580552 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-data\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.580861 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.580596 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-workload-certs\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.580861 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.580629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-token\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.580861 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.580661 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vfwt\" (UniqueName: \"kubernetes.io/projected/05fa6cec-639f-4ac7-8237-2edefd161dc7-kube-api-access-9vfwt\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.580861 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.580820 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-credential-socket\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.581051 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.580970 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-workload-certs\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.581428 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.581398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fa6cec-639f-4ac7-8237-2edefd161dc7-istiod-ca-cert\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.581561 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.581437 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-data\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.581561 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.581454 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-workload-socket\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.582935 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.582914 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-envoy\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.583199 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.583180 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-podinfo\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.592862 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.592827 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-token\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.592995 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.592920 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vfwt\" (UniqueName: \"kubernetes.io/projected/05fa6cec-639f-4ac7-8237-2edefd161dc7-kube-api-access-9vfwt\") pod \"e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.690863 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.690766 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:15.787715 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.787668 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:15.840741 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.840711 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g"] Jul 10 17:23:15.843246 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:23:15.843212 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fa6cec_639f_4ac7_8237_2edefd161dc7.slice/crio-14962691d49b8589a3c0bae391ce0c78f6c3cb89edd70f5e55804c1c418b8af1 WatchSource:0}: Error finding container 14962691d49b8589a3c0bae391ce0c78f6c3cb89edd70f5e55804c1c418b8af1: Status 404 returned error can't find the container with id 14962691d49b8589a3c0bae391ce0c78f6c3cb89edd70f5e55804c1c418b8af1 Jul 10 17:23:15.845296 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.845265 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:23:15.845377 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.845336 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:23:15.845422 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:15.845383 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:23:16.495861 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:16.495824 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" event={"ID":"05fa6cec-639f-4ac7-8237-2edefd161dc7","Type":"ContainerStarted","Data":"72f3d442887ffa29aa4dcabf133002c56ae1fd31f59b9e5eb58ec013e297920f"} Jul 10 17:23:16.496043 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:16.495886 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" event={"ID":"05fa6cec-639f-4ac7-8237-2edefd161dc7","Type":"ContainerStarted","Data":"14962691d49b8589a3c0bae391ce0c78f6c3cb89edd70f5e55804c1c418b8af1"} Jul 10 17:23:16.520209 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:16.520156 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" podStartSLOduration=1.520133313 podStartE2EDuration="1.520133313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:23:16.517533772 +0000 UTC m=+1736.643242182" watchObservedRunningTime="2026-07-10 17:23:16.520133313 +0000 UTC m=+1736.645841700" Jul 10 17:23:16.691458 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:16.691416 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:16.696412 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:16.696379 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:17.500833 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:17.500791 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:17.502252 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:17.502227 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:17.840403 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:17.840368 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:23:18.558178 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:18.558143 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:18.671026 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:18.670996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:18.804743 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:18.804703 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:19.008842 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:19.008807 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:19.332220 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:19.332126 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:19.438214 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:19.438180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:23.139849 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:23.139776 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:23.890759 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:23.890726 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:23.989079 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:23.989037 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:24.412074 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:24.412041 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:27.946661 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:27.946627 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:23:30.094697 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:30.094661 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:30.172956 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:30.172921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:30.556944 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:30.556863 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:30.909122 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:30.909082 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:30.969356 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:30.969316 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:31.579070 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:31.579038 2559 generic.go:356] "Generic (PLEG): container finished" podID="05fa6cec-639f-4ac7-8237-2edefd161dc7" containerID="72f3d442887ffa29aa4dcabf133002c56ae1fd31f59b9e5eb58ec013e297920f" exitCode=0 Jul 10 17:23:31.579518 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:31.579116 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" event={"ID":"05fa6cec-639f-4ac7-8237-2edefd161dc7","Type":"ContainerDied","Data":"72f3d442887ffa29aa4dcabf133002c56ae1fd31f59b9e5eb58ec013e297920f"} Jul 10 17:23:31.579580 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:31.579562 2559 scope.go:117] "RemoveContainer" containerID="72f3d442887ffa29aa4dcabf133002c56ae1fd31f59b9e5eb58ec013e297920f" Jul 10 17:23:31.580634 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:31.580603 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:23:31.580743 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:31.580713 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:23:31.580793 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:31.580747 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:23:32.501114 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:32.501070 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:32.591921 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:32.591875 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" event={"ID":"05fa6cec-639f-4ac7-8237-2edefd161dc7","Type":"ContainerStarted","Data":"69001fc6af09a38644f613b6b38b8d6b30e560616becfd3c3544dfaa75e3c49f"} Jul 10 17:23:32.692796 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:32.692752 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:32.693007 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:32.692932 2559 patch_prober.go:28] interesting pod/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 10 17:23:32.693007 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:32.692978 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" podUID="05fa6cec-639f-4ac7-8237-2edefd161dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 10 17:23:33.692167 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:33.692114 2559 patch_prober.go:28] interesting pod/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 10 17:23:33.692698 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:33.692193 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" podUID="05fa6cec-639f-4ac7-8237-2edefd161dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 10 17:23:34.691745 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:34.691699 2559 patch_prober.go:28] interesting pod/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 10 17:23:34.691953 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:34.691765 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" podUID="05fa6cec-639f-4ac7-8237-2edefd161dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 10 17:23:34.698767 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:34.698729 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:34.804359 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:34.804323 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:35.691606 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:35.691562 2559 patch_prober.go:28] interesting pod/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 10 17:23:35.691827 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:35.691646 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" podUID="05fa6cec-639f-4ac7-8237-2edefd161dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 10 17:23:36.695800 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:36.695761 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:36.696173 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:36.696037 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:36.696701 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:36.696685 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:36.942948 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:36.942903 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:37.042905 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:37.042806 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:37.214074 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:37.214036 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g"] Jul 10 17:23:37.239318 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:37.239288 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:37.526655 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:37.526622 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:37.623486 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:37.623454 2559 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" secret="" err="secret \"e2e-mt-3ff7ed0c-openshift-default-dockercfg-wlr5c\" not found" Jul 10 17:23:37.685133 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:37.685096 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:37.788384 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:37.788273 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:37.955405 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:37.955375 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:38.628948 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:38.628911 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" podUID="05fa6cec-639f-4ac7-8237-2edefd161dc7" containerName="istio-proxy" containerID="cri-o://69001fc6af09a38644f613b6b38b8d6b30e560616becfd3c3544dfaa75e3c49f" gracePeriod=30 Jul 10 17:23:39.657163 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:39.657130 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:23:43.884449 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.884424 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:43.946710 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.946630 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-envoy\") pod \"05fa6cec-639f-4ac7-8237-2edefd161dc7\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " Jul 10 17:23:43.946710 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.946677 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-podinfo\") pod \"05fa6cec-639f-4ac7-8237-2edefd161dc7\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " Jul 10 17:23:43.946903 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.946746 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fa6cec-639f-4ac7-8237-2edefd161dc7-istiod-ca-cert\") pod \"05fa6cec-639f-4ac7-8237-2edefd161dc7\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " Jul 10 17:23:43.946903 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.946766 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-workload-certs\") pod \"05fa6cec-639f-4ac7-8237-2edefd161dc7\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " Jul 10 17:23:43.946903 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.946848 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vfwt\" (UniqueName: \"kubernetes.io/projected/05fa6cec-639f-4ac7-8237-2edefd161dc7-kube-api-access-9vfwt\") pod \"05fa6cec-639f-4ac7-8237-2edefd161dc7\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " Jul 10 17:23:43.946903 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.946894 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-data\") pod \"05fa6cec-639f-4ac7-8237-2edefd161dc7\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " Jul 10 17:23:43.947110 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.946916 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-token\") pod \"05fa6cec-639f-4ac7-8237-2edefd161dc7\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " Jul 10 17:23:43.947110 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.946942 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-credential-socket\") pod \"05fa6cec-639f-4ac7-8237-2edefd161dc7\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " Jul 10 17:23:43.947110 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.946970 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-workload-socket\") pod \"05fa6cec-639f-4ac7-8237-2edefd161dc7\" (UID: \"05fa6cec-639f-4ac7-8237-2edefd161dc7\") " Jul 10 17:23:43.947110 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.947061 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "05fa6cec-639f-4ac7-8237-2edefd161dc7" (UID: "05fa6cec-639f-4ac7-8237-2edefd161dc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:23:43.947328 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.947165 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05fa6cec-639f-4ac7-8237-2edefd161dc7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "05fa6cec-639f-4ac7-8237-2edefd161dc7" (UID: "05fa6cec-639f-4ac7-8237-2edefd161dc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:23:43.947328 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.947246 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "05fa6cec-639f-4ac7-8237-2edefd161dc7" (UID: "05fa6cec-639f-4ac7-8237-2edefd161dc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:23:43.947442 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.947428 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "05fa6cec-639f-4ac7-8237-2edefd161dc7" (UID: "05fa6cec-639f-4ac7-8237-2edefd161dc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:23:43.947535 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.947419 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fa6cec-639f-4ac7-8237-2edefd161dc7-istiod-ca-cert\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:23:43.947760 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.947616 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-workload-certs\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:23:43.947760 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.947631 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-workload-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:23:43.947998 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.947962 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-data" (OuterVolumeSpecName: "istio-data") pod "05fa6cec-639f-4ac7-8237-2edefd161dc7" (UID: "05fa6cec-639f-4ac7-8237-2edefd161dc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:23:43.949099 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.949069 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "05fa6cec-639f-4ac7-8237-2edefd161dc7" (UID: "05fa6cec-639f-4ac7-8237-2edefd161dc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:23:43.949099 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.949073 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "05fa6cec-639f-4ac7-8237-2edefd161dc7" (UID: "05fa6cec-639f-4ac7-8237-2edefd161dc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:23:43.949260 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.949216 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fa6cec-639f-4ac7-8237-2edefd161dc7-kube-api-access-9vfwt" (OuterVolumeSpecName: "kube-api-access-9vfwt") pod "05fa6cec-639f-4ac7-8237-2edefd161dc7" (UID: "05fa6cec-639f-4ac7-8237-2edefd161dc7"). InnerVolumeSpecName "kube-api-access-9vfwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:23:43.949260 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:43.949248 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-token" (OuterVolumeSpecName: "istio-token") pod "05fa6cec-639f-4ac7-8237-2edefd161dc7" (UID: "05fa6cec-639f-4ac7-8237-2edefd161dc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:23:44.009264 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.009226 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:44.048549 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.048485 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vfwt\" (UniqueName: \"kubernetes.io/projected/05fa6cec-639f-4ac7-8237-2edefd161dc7-kube-api-access-9vfwt\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:23:44.048549 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.048545 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-data\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:23:44.048549 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.048558 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-token\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:23:44.048785 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.048570 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-credential-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:23:44.048785 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.048581 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-envoy\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:23:44.048785 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.048593 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fa6cec-639f-4ac7-8237-2edefd161dc7-istio-podinfo\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:23:44.164638 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.164585 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:44.299460 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.299375 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:44.461129 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.461091 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:44.662385 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.662346 2559 generic.go:356] "Generic (PLEG): container finished" podID="05fa6cec-639f-4ac7-8237-2edefd161dc7" containerID="69001fc6af09a38644f613b6b38b8d6b30e560616becfd3c3544dfaa75e3c49f" exitCode=0 Jul 10 17:23:44.662624 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.662393 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" Jul 10 17:23:44.662624 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.662416 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" event={"ID":"05fa6cec-639f-4ac7-8237-2edefd161dc7","Type":"ContainerDied","Data":"69001fc6af09a38644f613b6b38b8d6b30e560616becfd3c3544dfaa75e3c49f"} Jul 10 17:23:44.662624 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.662471 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g" event={"ID":"05fa6cec-639f-4ac7-8237-2edefd161dc7","Type":"ContainerDied","Data":"14962691d49b8589a3c0bae391ce0c78f6c3cb89edd70f5e55804c1c418b8af1"} Jul 10 17:23:44.662624 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.662536 2559 scope.go:117] "RemoveContainer" containerID="69001fc6af09a38644f613b6b38b8d6b30e560616becfd3c3544dfaa75e3c49f" Jul 10 17:23:44.671839 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.671820 2559 scope.go:117] "RemoveContainer" containerID="72f3d442887ffa29aa4dcabf133002c56ae1fd31f59b9e5eb58ec013e297920f" Jul 10 17:23:44.680576 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.680551 2559 scope.go:117] "RemoveContainer" containerID="69001fc6af09a38644f613b6b38b8d6b30e560616becfd3c3544dfaa75e3c49f" Jul 10 17:23:44.680877 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:23:44.680857 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69001fc6af09a38644f613b6b38b8d6b30e560616becfd3c3544dfaa75e3c49f\": container with ID starting with 69001fc6af09a38644f613b6b38b8d6b30e560616becfd3c3544dfaa75e3c49f not found: ID does not exist" containerID="69001fc6af09a38644f613b6b38b8d6b30e560616becfd3c3544dfaa75e3c49f" Jul 10 17:23:44.680946 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.680888 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69001fc6af09a38644f613b6b38b8d6b30e560616becfd3c3544dfaa75e3c49f"} err="failed to get container status \"69001fc6af09a38644f613b6b38b8d6b30e560616becfd3c3544dfaa75e3c49f\": rpc error: code = NotFound desc = could not find container \"69001fc6af09a38644f613b6b38b8d6b30e560616becfd3c3544dfaa75e3c49f\": container with ID starting with 69001fc6af09a38644f613b6b38b8d6b30e560616becfd3c3544dfaa75e3c49f not found: ID does not exist" Jul 10 17:23:44.680946 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.680907 2559 scope.go:117] "RemoveContainer" containerID="72f3d442887ffa29aa4dcabf133002c56ae1fd31f59b9e5eb58ec013e297920f" Jul 10 17:23:44.681125 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:23:44.681107 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f3d442887ffa29aa4dcabf133002c56ae1fd31f59b9e5eb58ec013e297920f\": container with ID starting with 72f3d442887ffa29aa4dcabf133002c56ae1fd31f59b9e5eb58ec013e297920f not found: ID does not exist" containerID="72f3d442887ffa29aa4dcabf133002c56ae1fd31f59b9e5eb58ec013e297920f" Jul 10 17:23:44.681164 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.681130 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f3d442887ffa29aa4dcabf133002c56ae1fd31f59b9e5eb58ec013e297920f"} err="failed to get container status \"72f3d442887ffa29aa4dcabf133002c56ae1fd31f59b9e5eb58ec013e297920f\": rpc error: code = NotFound desc = could not find container \"72f3d442887ffa29aa4dcabf133002c56ae1fd31f59b9e5eb58ec013e297920f\": container with ID starting with 72f3d442887ffa29aa4dcabf133002c56ae1fd31f59b9e5eb58ec013e297920f not found: ID does not exist" Jul 10 17:23:44.689447 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.689419 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g"] Jul 10 17:23:44.705589 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:44.705562 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3ff7ed0c-openshift-default-76c97cb96c-2gr8g"] Jul 10 17:23:46.398886 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:46.398845 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05fa6cec-639f-4ac7-8237-2edefd161dc7" path="/var/lib/kubelet/pods/05fa6cec-639f-4ac7-8237-2edefd161dc7/volumes" Jul 10 17:23:51.101676 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:51.101645 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:51.205870 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:51.205836 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:58.167630 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.167596 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:58.511553 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.511447 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k"] Jul 10 17:23:58.512055 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.512038 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05fa6cec-639f-4ac7-8237-2edefd161dc7" containerName="istio-proxy" Jul 10 17:23:58.512117 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.512057 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="05fa6cec-639f-4ac7-8237-2edefd161dc7" containerName="istio-proxy" Jul 10 17:23:58.512165 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.512154 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="05fa6cec-639f-4ac7-8237-2edefd161dc7" containerName="istio-proxy" Jul 10 17:23:58.512206 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.512172 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="05fa6cec-639f-4ac7-8237-2edefd161dc7" containerName="istio-proxy" Jul 10 17:23:58.515680 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.515660 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.518463 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.518435 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d9b2224f-openshift-default-dockercfg-4r789\"" Jul 10 17:23:58.527879 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.527854 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:58.530278 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.530248 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k"] Jul 10 17:23:58.683105 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.683071 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba126d6-ace5-487f-9992-e3067a051093-istio-token\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.683309 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.683117 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-credential-socket\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.683309 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.683144 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba126d6-ace5-487f-9992-e3067a051093-istio-podinfo\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.683309 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.683229 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-istio-data\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.683309 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.683267 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba126d6-ace5-487f-9992-e3067a051093-istiod-ca-cert\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.683539 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.683346 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-workload-certs\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.683539 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.683409 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-istio-envoy\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.683539 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.683432 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psp95\" (UniqueName: \"kubernetes.io/projected/1ba126d6-ace5-487f-9992-e3067a051093-kube-api-access-psp95\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.683539 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.683479 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-workload-socket\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.784272 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.784167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-workload-certs\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.784272 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.784231 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-istio-envoy\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.784272 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.784261 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psp95\" (UniqueName: \"kubernetes.io/projected/1ba126d6-ace5-487f-9992-e3067a051093-kube-api-access-psp95\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.784625 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.784300 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-workload-socket\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.784625 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.784344 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba126d6-ace5-487f-9992-e3067a051093-istio-token\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.784625 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.784383 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-credential-socket\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.784625 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.784550 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba126d6-ace5-487f-9992-e3067a051093-istio-podinfo\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.784625 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.784617 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-istio-data\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.784886 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.784652 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba126d6-ace5-487f-9992-e3067a051093-istiod-ca-cert\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.784886 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.784667 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-workload-certs\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.784992 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.784946 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-credential-socket\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.785192 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.785167 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-workload-socket\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.785284 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.785215 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-istio-data\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.785342 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.785306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba126d6-ace5-487f-9992-e3067a051093-istiod-ca-cert\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.787352 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.787328 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-istio-envoy\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.787687 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.787663 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba126d6-ace5-487f-9992-e3067a051093-istio-podinfo\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.798067 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.798037 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psp95\" (UniqueName: \"kubernetes.io/projected/1ba126d6-ace5-487f-9992-e3067a051093-kube-api-access-psp95\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.798996 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.798977 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba126d6-ace5-487f-9992-e3067a051093-istio-token\") pod \"e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.829183 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.829148 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:23:58.998969 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.998938 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k"] Jul 10 17:23:58.999171 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:58.999147 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:59.000263 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:23:59.000238 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba126d6_ace5_487f_9992_e3067a051093.slice/crio-c7dc6b8974af6d7118869b99bb2702a6fa8d2edcd1f01e2ce57549c6a135e1d8 WatchSource:0}: Error finding container c7dc6b8974af6d7118869b99bb2702a6fa8d2edcd1f01e2ce57549c6a135e1d8: Status 404 returned error can't find the container with id c7dc6b8974af6d7118869b99bb2702a6fa8d2edcd1f01e2ce57549c6a135e1d8 Jul 10 17:23:59.002245 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:59.002210 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:23:59.002315 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:59.002278 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:23:59.002351 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:59.002311 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:23:59.742909 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:59.742874 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" event={"ID":"1ba126d6-ace5-487f-9992-e3067a051093","Type":"ContainerStarted","Data":"485b44c253af58ac0b68a37cbac74c696754915c6d5c9eec5c6efe79c0a8dbdd"} Jul 10 17:23:59.743298 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:59.742933 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" event={"ID":"1ba126d6-ace5-487f-9992-e3067a051093","Type":"ContainerStarted","Data":"c7dc6b8974af6d7118869b99bb2702a6fa8d2edcd1f01e2ce57549c6a135e1d8"} Jul 10 17:23:59.768338 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:59.768278 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" podStartSLOduration=1.7682611719999999 podStartE2EDuration="1.768261172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:23:59.765344732 +0000 UTC m=+1779.891053135" watchObservedRunningTime="2026-07-10 17:23:59.768261172 +0000 UTC m=+1779.893969594" Jul 10 17:23:59.829770 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:23:59.829725 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:24:00.835613 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:00.835581 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:24:01.755008 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:01.754976 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:24:01.756143 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:01.756122 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:24:01.840181 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:01.840147 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:01.938125 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:01.938089 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:02.074866 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:02.074773 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:02.639236 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:02.639201 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:02.745202 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:02.745166 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:06.353707 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:06.353666 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:07.751321 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:07.751283 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:24:15.843138 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:15.843106 2559 generic.go:356] "Generic (PLEG): container finished" podID="1ba126d6-ace5-487f-9992-e3067a051093" containerID="485b44c253af58ac0b68a37cbac74c696754915c6d5c9eec5c6efe79c0a8dbdd" exitCode=0 Jul 10 17:24:15.843605 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:15.843159 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" event={"ID":"1ba126d6-ace5-487f-9992-e3067a051093","Type":"ContainerDied","Data":"485b44c253af58ac0b68a37cbac74c696754915c6d5c9eec5c6efe79c0a8dbdd"} Jul 10 17:24:15.843605 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:15.843557 2559 scope.go:117] "RemoveContainer" containerID="485b44c253af58ac0b68a37cbac74c696754915c6d5c9eec5c6efe79c0a8dbdd" Jul 10 17:24:15.844598 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:15.844570 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:24:15.844666 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:15.844635 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:24:15.844702 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:15.844664 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:24:16.754990 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:16.754951 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:24:16.850300 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:16.850263 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" event={"ID":"1ba126d6-ace5-487f-9992-e3067a051093","Type":"ContainerStarted","Data":"e338a6465baa27e218e50427a19ec1265da469ffc73e225aff59cb0b7edb62d6"} Jul 10 17:24:17.829846 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:17.829813 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:24:17.830883 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:17.830862 2559 patch_prober.go:28] interesting pod/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 10 17:24:17.830982 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:17.830906 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" podUID="1ba126d6-ace5-487f-9992-e3067a051093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 10 17:24:18.830280 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:18.830236 2559 patch_prober.go:28] interesting pod/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 10 17:24:18.830726 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:18.830303 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" podUID="1ba126d6-ace5-487f-9992-e3067a051093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 10 17:24:19.829761 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.829723 2559 patch_prober.go:28] interesting pod/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 10 17:24:19.829944 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.829781 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" podUID="1ba126d6-ace5-487f-9992-e3067a051093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 10 17:24:19.863244 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.863208 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m"] Jul 10 17:24:19.863642 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.863628 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05fa6cec-639f-4ac7-8237-2edefd161dc7" containerName="istio-proxy" Jul 10 17:24:19.863697 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.863644 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="05fa6cec-639f-4ac7-8237-2edefd161dc7" containerName="istio-proxy" Jul 10 17:24:19.866284 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.866254 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:19.870215 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.870192 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d9b2224f\"/\"kube-root-ca.crt\"" Jul 10 17:24:19.870354 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.870220 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d9b2224f\"/\"default-dockercfg-rbd2k\"" Jul 10 17:24:19.870354 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.870237 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d9b2224f\"/\"openshift-service-ca.crt\"" Jul 10 17:24:19.870354 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.870242 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d9b2224f\"/\"e2e-lifecycle-model-d9b2224f-kserve-self-signed-certs\"" Jul 10 17:24:19.876083 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.876059 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m"] Jul 10 17:24:19.906518 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.899722 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-home\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:19.906518 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.899771 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-dshm\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:19.906518 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.899810 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92nf6\" (UniqueName: \"kubernetes.io/projected/a4667e2a-2a20-4509-af32-e4c74c13c37d-kube-api-access-92nf6\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:19.906518 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.899843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4667e2a-2a20-4509-af32-e4c74c13c37d-tls-certs\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:19.906518 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.899941 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-model-cache\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:19.906518 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.899979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-tmp-dir\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:19.906518 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:19.900969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:20.001319 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.001275 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-home\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:20.001569 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.001336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-dshm\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:20.001569 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.001384 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92nf6\" (UniqueName: \"kubernetes.io/projected/a4667e2a-2a20-4509-af32-e4c74c13c37d-kube-api-access-92nf6\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:20.001569 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.001418 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4667e2a-2a20-4509-af32-e4c74c13c37d-tls-certs\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:20.001569 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.001543 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-model-cache\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:20.001820 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.001584 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-tmp-dir\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:20.002315 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.002035 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-tmp-dir\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:20.002315 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.002287 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-model-cache\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:20.002529 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.002437 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-home\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:20.006110 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.006059 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4667e2a-2a20-4509-af32-e4c74c13c37d-tls-certs\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:20.006984 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.006950 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-dshm\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:20.011239 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.011211 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92nf6\" (UniqueName: \"kubernetes.io/projected/a4667e2a-2a20-4509-af32-e4c74c13c37d-kube-api-access-92nf6\") pod \"e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:20.176878 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.176836 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:20.370658 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.370626 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m"] Jul 10 17:24:20.371878 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:24:20.371850 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4667e2a_2a20_4509_af32_e4c74c13c37d.slice/crio-9a61ca7725b2520dfa748dad6917c8e9458301641eb92af581881e535d28b4b1 WatchSource:0}: Error finding container 9a61ca7725b2520dfa748dad6917c8e9458301641eb92af581881e535d28b4b1: Status 404 returned error can't find the container with id 9a61ca7725b2520dfa748dad6917c8e9458301641eb92af581881e535d28b4b1 Jul 10 17:24:20.834372 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.834295 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:24:20.834769 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.834748 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:24:20.835341 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.835317 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:24:20.874224 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.874183 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" event={"ID":"a4667e2a-2a20-4509-af32-e4c74c13c37d","Type":"ContainerStarted","Data":"51ee3300833bd65d6299e3c765b2e6ba19afadc51534ec23bfec55f4a018b4fc"} Jul 10 17:24:20.874224 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.874225 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" event={"ID":"a4667e2a-2a20-4509-af32-e4c74c13c37d","Type":"ContainerStarted","Data":"9a61ca7725b2520dfa748dad6917c8e9458301641eb92af581881e535d28b4b1"} Jul 10 17:24:20.892587 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:20.892516 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" podStartSLOduration=1.8924833859999999 podStartE2EDuration="1.892483386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:24:20.890661103 +0000 UTC m=+1801.016369513" watchObservedRunningTime="2026-07-10 17:24:20.892483386 +0000 UTC m=+1801.018191796" Jul 10 17:24:23.134701 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:23.134661 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:24:28.403119 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:28.403078 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:30.177279 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:30.177241 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:30.177279 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:30.177290 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:30.190054 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:30.190022 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:30.945224 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:30.945193 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:24:38.535642 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:38.535604 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:24:44.039549 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:44.039482 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:44.377680 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:44.377649 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:49.134392 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:49.134356 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:24:50.266777 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:50.266741 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:55.246010 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:55.245972 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m"] Jul 10 17:24:55.341825 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:55.341790 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:59.531364 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:24:59.531318 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:25:10.267878 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.267813 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" podUID="a4667e2a-2a20-4509-af32-e4c74c13c37d" containerName="main" containerID="cri-o://51ee3300833bd65d6299e3c765b2e6ba19afadc51534ec23bfec55f4a018b4fc" gracePeriod=45 Jul 10 17:25:10.534995 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.534966 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:25:10.671455 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.671423 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-model-cache\") pod \"a4667e2a-2a20-4509-af32-e4c74c13c37d\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " Jul 10 17:25:10.671696 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.671513 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92nf6\" (UniqueName: \"kubernetes.io/projected/a4667e2a-2a20-4509-af32-e4c74c13c37d-kube-api-access-92nf6\") pod \"a4667e2a-2a20-4509-af32-e4c74c13c37d\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " Jul 10 17:25:10.671696 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.671538 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-home\") pod \"a4667e2a-2a20-4509-af32-e4c74c13c37d\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " Jul 10 17:25:10.671696 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.671591 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-tmp-dir\") pod \"a4667e2a-2a20-4509-af32-e4c74c13c37d\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " Jul 10 17:25:10.671873 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.671716 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-dshm\") pod \"a4667e2a-2a20-4509-af32-e4c74c13c37d\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " Jul 10 17:25:10.671873 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.671754 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4667e2a-2a20-4509-af32-e4c74c13c37d-tls-certs\") pod \"a4667e2a-2a20-4509-af32-e4c74c13c37d\" (UID: \"a4667e2a-2a20-4509-af32-e4c74c13c37d\") " Jul 10 17:25:10.671873 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.671710 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4667e2a-2a20-4509-af32-e4c74c13c37d" (UID: "a4667e2a-2a20-4509-af32-e4c74c13c37d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:10.672026 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.671871 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4667e2a-2a20-4509-af32-e4c74c13c37d" (UID: "a4667e2a-2a20-4509-af32-e4c74c13c37d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:10.672026 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.671887 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-home" (OuterVolumeSpecName: "home") pod "a4667e2a-2a20-4509-af32-e4c74c13c37d" (UID: "a4667e2a-2a20-4509-af32-e4c74c13c37d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:10.672123 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.672091 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-tmp-dir\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:10.672123 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.672113 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-model-cache\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:10.672222 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.672125 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-home\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:10.674094 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.674069 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-dshm" (OuterVolumeSpecName: "dshm") pod "a4667e2a-2a20-4509-af32-e4c74c13c37d" (UID: "a4667e2a-2a20-4509-af32-e4c74c13c37d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:10.674221 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.674097 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4667e2a-2a20-4509-af32-e4c74c13c37d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4667e2a-2a20-4509-af32-e4c74c13c37d" (UID: "a4667e2a-2a20-4509-af32-e4c74c13c37d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:25:10.674221 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.674133 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4667e2a-2a20-4509-af32-e4c74c13c37d-kube-api-access-92nf6" (OuterVolumeSpecName: "kube-api-access-92nf6") pod "a4667e2a-2a20-4509-af32-e4c74c13c37d" (UID: "a4667e2a-2a20-4509-af32-e4c74c13c37d"). InnerVolumeSpecName "kube-api-access-92nf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:25:10.773301 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.773208 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92nf6\" (UniqueName: \"kubernetes.io/projected/a4667e2a-2a20-4509-af32-e4c74c13c37d-kube-api-access-92nf6\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:10.773301 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.773243 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4667e2a-2a20-4509-af32-e4c74c13c37d-dshm\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:10.773301 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:10.773255 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4667e2a-2a20-4509-af32-e4c74c13c37d-tls-certs\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:11.161701 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:11.161657 2559 generic.go:356] "Generic (PLEG): container finished" podID="a4667e2a-2a20-4509-af32-e4c74c13c37d" containerID="51ee3300833bd65d6299e3c765b2e6ba19afadc51534ec23bfec55f4a018b4fc" exitCode=0 Jul 10 17:25:11.161867 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:11.161708 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" event={"ID":"a4667e2a-2a20-4509-af32-e4c74c13c37d","Type":"ContainerDied","Data":"51ee3300833bd65d6299e3c765b2e6ba19afadc51534ec23bfec55f4a018b4fc"} Jul 10 17:25:11.161867 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:11.161754 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" event={"ID":"a4667e2a-2a20-4509-af32-e4c74c13c37d","Type":"ContainerDied","Data":"9a61ca7725b2520dfa748dad6917c8e9458301641eb92af581881e535d28b4b1"} Jul 10 17:25:11.161867 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:11.161766 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m" Jul 10 17:25:11.161867 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:11.161782 2559 scope.go:117] "RemoveContainer" containerID="51ee3300833bd65d6299e3c765b2e6ba19afadc51534ec23bfec55f4a018b4fc" Jul 10 17:25:11.171792 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:11.171766 2559 scope.go:117] "RemoveContainer" containerID="51ee3300833bd65d6299e3c765b2e6ba19afadc51534ec23bfec55f4a018b4fc" Jul 10 17:25:11.172062 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:25:11.172040 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ee3300833bd65d6299e3c765b2e6ba19afadc51534ec23bfec55f4a018b4fc\": container with ID starting with 51ee3300833bd65d6299e3c765b2e6ba19afadc51534ec23bfec55f4a018b4fc not found: ID does not exist" containerID="51ee3300833bd65d6299e3c765b2e6ba19afadc51534ec23bfec55f4a018b4fc" Jul 10 17:25:11.172121 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:11.172080 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ee3300833bd65d6299e3c765b2e6ba19afadc51534ec23bfec55f4a018b4fc"} err="failed to get container status \"51ee3300833bd65d6299e3c765b2e6ba19afadc51534ec23bfec55f4a018b4fc\": rpc error: code = NotFound desc = could not find container \"51ee3300833bd65d6299e3c765b2e6ba19afadc51534ec23bfec55f4a018b4fc\": container with ID starting with 51ee3300833bd65d6299e3c765b2e6ba19afadc51534ec23bfec55f4a018b4fc not found: ID does not exist" Jul 10 17:25:11.184115 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:11.184083 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m"] Jul 10 17:25:11.187841 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:11.187810 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d9b2224f/e2e-lifecycle-model-d9b2224f-kserve-5b6d7d9879-h5r7m"] Jul 10 17:25:12.399749 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:12.399717 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4667e2a-2a20-4509-af32-e4c74c13c37d" path="/var/lib/kubelet/pods/a4667e2a-2a20-4509-af32-e4c74c13c37d/volumes" Jul 10 17:25:31.963957 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:31.963915 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:32.357109 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:32.357073 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:35.156199 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:35.156162 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:25:37.921394 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:37.921355 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:37.986503 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:37.986463 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:38.215723 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:38.215620 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k"] Jul 10 17:25:38.215949 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:38.215919 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" podUID="1ba126d6-ace5-487f-9992-e3067a051093" containerName="istio-proxy" containerID="cri-o://e338a6465baa27e218e50427a19ec1265da469ffc73e225aff59cb0b7edb62d6" gracePeriod=30 Jul 10 17:25:38.230675 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:38.230643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:38.477843 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:38.477752 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:39.622376 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:39.622336 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:39.721397 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:39.721365 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:39.821211 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:39.821179 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:40.669203 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:40.669126 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:40.769158 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:40.769119 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:40.869300 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:40.869271 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:42.379719 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:42.379688 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:43.341134 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.341101 2559 generic.go:356] "Generic (PLEG): container finished" podID="1ba126d6-ace5-487f-9992-e3067a051093" containerID="e338a6465baa27e218e50427a19ec1265da469ffc73e225aff59cb0b7edb62d6" exitCode=0 Jul 10 17:25:43.341334 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.341177 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" event={"ID":"1ba126d6-ace5-487f-9992-e3067a051093","Type":"ContainerDied","Data":"e338a6465baa27e218e50427a19ec1265da469ffc73e225aff59cb0b7edb62d6"} Jul 10 17:25:43.341334 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.341277 2559 scope.go:117] "RemoveContainer" containerID="485b44c253af58ac0b68a37cbac74c696754915c6d5c9eec5c6efe79c0a8dbdd" Jul 10 17:25:43.477077 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.477000 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:25:43.589668 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.589624 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba126d6-ace5-487f-9992-e3067a051093-istio-token\") pod \"1ba126d6-ace5-487f-9992-e3067a051093\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " Jul 10 17:25:43.589848 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.589683 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba126d6-ace5-487f-9992-e3067a051093-istiod-ca-cert\") pod \"1ba126d6-ace5-487f-9992-e3067a051093\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " Jul 10 17:25:43.589848 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.589708 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-istio-envoy\") pod \"1ba126d6-ace5-487f-9992-e3067a051093\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " Jul 10 17:25:43.589848 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.589733 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba126d6-ace5-487f-9992-e3067a051093-istio-podinfo\") pod \"1ba126d6-ace5-487f-9992-e3067a051093\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " Jul 10 17:25:43.589848 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.589816 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-workload-socket\") pod \"1ba126d6-ace5-487f-9992-e3067a051093\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " Jul 10 17:25:43.590047 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.589861 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-istio-data\") pod \"1ba126d6-ace5-487f-9992-e3067a051093\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " Jul 10 17:25:43.590047 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.589898 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-credential-socket\") pod \"1ba126d6-ace5-487f-9992-e3067a051093\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " Jul 10 17:25:43.590047 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.589923 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psp95\" (UniqueName: \"kubernetes.io/projected/1ba126d6-ace5-487f-9992-e3067a051093-kube-api-access-psp95\") pod \"1ba126d6-ace5-487f-9992-e3067a051093\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " Jul 10 17:25:43.590047 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.589964 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-workload-certs\") pod \"1ba126d6-ace5-487f-9992-e3067a051093\" (UID: \"1ba126d6-ace5-487f-9992-e3067a051093\") " Jul 10 17:25:43.590251 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.590156 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba126d6-ace5-487f-9992-e3067a051093-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ba126d6-ace5-487f-9992-e3067a051093" (UID: "1ba126d6-ace5-487f-9992-e3067a051093"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:25:43.590313 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.590259 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ba126d6-ace5-487f-9992-e3067a051093" (UID: "1ba126d6-ace5-487f-9992-e3067a051093"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:43.590383 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.590352 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ba126d6-ace5-487f-9992-e3067a051093" (UID: "1ba126d6-ace5-487f-9992-e3067a051093"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:43.590719 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.590691 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ba126d6-ace5-487f-9992-e3067a051093" (UID: "1ba126d6-ace5-487f-9992-e3067a051093"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:43.591020 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.590794 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba126d6-ace5-487f-9992-e3067a051093-istiod-ca-cert\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:43.591020 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.590813 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-workload-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:43.591020 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.590826 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-credential-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:43.591020 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.590840 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-workload-certs\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:43.591511 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.591466 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ba126d6-ace5-487f-9992-e3067a051093" (UID: "1ba126d6-ace5-487f-9992-e3067a051093"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:43.592433 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.592405 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ba126d6-ace5-487f-9992-e3067a051093-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ba126d6-ace5-487f-9992-e3067a051093" (UID: "1ba126d6-ace5-487f-9992-e3067a051093"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:25:43.592537 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.592517 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba126d6-ace5-487f-9992-e3067a051093-kube-api-access-psp95" (OuterVolumeSpecName: "kube-api-access-psp95") pod "1ba126d6-ace5-487f-9992-e3067a051093" (UID: "1ba126d6-ace5-487f-9992-e3067a051093"). InnerVolumeSpecName "kube-api-access-psp95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:25:43.592669 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.592647 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ba126d6-ace5-487f-9992-e3067a051093" (UID: "1ba126d6-ace5-487f-9992-e3067a051093"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:43.592730 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.592649 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba126d6-ace5-487f-9992-e3067a051093-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ba126d6-ace5-487f-9992-e3067a051093" (UID: "1ba126d6-ace5-487f-9992-e3067a051093"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:25:43.691428 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.691390 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba126d6-ace5-487f-9992-e3067a051093-istio-token\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:43.691428 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.691425 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-istio-envoy\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:43.691428 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.691437 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba126d6-ace5-487f-9992-e3067a051093-istio-podinfo\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:43.691714 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.691448 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba126d6-ace5-487f-9992-e3067a051093-istio-data\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:43.691714 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:43.691461 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psp95\" (UniqueName: \"kubernetes.io/projected/1ba126d6-ace5-487f-9992-e3067a051093-kube-api-access-psp95\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:25:44.348546 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:44.348488 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" Jul 10 17:25:44.348758 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:44.348559 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k" event={"ID":"1ba126d6-ace5-487f-9992-e3067a051093","Type":"ContainerDied","Data":"c7dc6b8974af6d7118869b99bb2702a6fa8d2edcd1f01e2ce57549c6a135e1d8"} Jul 10 17:25:44.348758 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:44.348637 2559 scope.go:117] "RemoveContainer" containerID="e338a6465baa27e218e50427a19ec1265da469ffc73e225aff59cb0b7edb62d6" Jul 10 17:25:44.370173 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:44.370142 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k"] Jul 10 17:25:44.376063 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:44.376031 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d9b2224f-openshift-default-db78fbf4c-lrv6k"] Jul 10 17:25:44.399318 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:44.399283 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ba126d6-ace5-487f-9992-e3067a051093" path="/var/lib/kubelet/pods/1ba126d6-ace5-487f-9992-e3067a051093/volumes" Jul 10 17:25:48.224837 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:48.224796 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:48.325151 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:48.325118 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:48.835689 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:48.835639 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:25:48.932234 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:48.932196 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:54.669721 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:54.669643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:54.777688 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:54.777654 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:54.876411 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:54.876376 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:25:58.734478 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:25:58.734439 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:26:05.444782 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:05.444750 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:06.756687 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:06.756652 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:26.868720 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:26.868680 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:32.438543 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:32.438487 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:32.502387 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:32.502356 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:32.828531 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:32.828430 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:33.169793 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:33.169760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:33.830175 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:33.830140 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:36.048003 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:36.047960 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:36.151058 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:36.151021 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:36.266235 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:36.266202 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:36.808782 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:36.808751 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:36.909758 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:36.909721 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:40.624404 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:40.624363 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:40.977890 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:40.977797 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:41.298723 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.298636 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4"] Jul 10 17:26:41.299228 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.299209 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba126d6-ace5-487f-9992-e3067a051093" containerName="istio-proxy" Jul 10 17:26:41.299272 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.299233 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba126d6-ace5-487f-9992-e3067a051093" containerName="istio-proxy" Jul 10 17:26:41.299272 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.299251 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba126d6-ace5-487f-9992-e3067a051093" containerName="istio-proxy" Jul 10 17:26:41.299272 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.299260 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba126d6-ace5-487f-9992-e3067a051093" containerName="istio-proxy" Jul 10 17:26:41.299374 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.299279 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4667e2a-2a20-4509-af32-e4c74c13c37d" containerName="main" Jul 10 17:26:41.299374 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.299288 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4667e2a-2a20-4509-af32-e4c74c13c37d" containerName="main" Jul 10 17:26:41.299436 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.299413 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ba126d6-ace5-487f-9992-e3067a051093" containerName="istio-proxy" Jul 10 17:26:41.299436 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.299425 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ba126d6-ace5-487f-9992-e3067a051093" containerName="istio-proxy" Jul 10 17:26:41.299519 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.299437 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4667e2a-2a20-4509-af32-e4c74c13c37d" containerName="main" Jul 10 17:26:41.302952 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.302925 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.305560 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.305538 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fc13d1-openshift-default-dockercfg-nh8wk\"" Jul 10 17:26:41.312775 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.312752 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4"] Jul 10 17:26:41.329824 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.329789 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:41.425585 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.425550 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-credential-socket\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.425585 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.425591 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istiod-ca-cert\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.425805 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.425611 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-workload-certs\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.425805 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.425678 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p95vd\" (UniqueName: \"kubernetes.io/projected/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-kube-api-access-p95vd\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.425805 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.425742 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-token\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.425805 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.425767 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-envoy\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.425932 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.425815 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-workload-socket\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.425932 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.425830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-data\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.425932 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.425900 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-podinfo\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.527270 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.527228 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-workload-socket\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.527270 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.527275 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-data\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.527547 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.527325 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-podinfo\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.527547 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.527376 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-credential-socket\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.527547 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.527399 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istiod-ca-cert\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.527547 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.527427 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-workload-certs\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.527547 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.527464 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p95vd\" (UniqueName: \"kubernetes.io/projected/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-kube-api-access-p95vd\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.527829 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.527553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-token\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.527829 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.527584 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-envoy\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.527829 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.527703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-workload-socket\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.527989 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.527862 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-data\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.528239 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.528219 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istiod-ca-cert\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.528332 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.528237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-credential-socket\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.528578 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.528556 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-workload-certs\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.529987 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.529966 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-envoy\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.530137 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.530118 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-podinfo\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.535105 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.535084 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-token\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.535577 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.535554 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p95vd\" (UniqueName: \"kubernetes.io/projected/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-kube-api-access-p95vd\") pod \"e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.615525 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.615482 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:41.633685 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.633655 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:41.760171 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.760144 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4"] Jul 10 17:26:41.761679 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:26:41.761641 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d4cd98_d80e_421c_b91e_7558bf2dae5a.slice/crio-26922dac3bbc7846bdbaf5749b06922e8101cf6331d79e29a562816fd29640ec WatchSource:0}: Error finding container 26922dac3bbc7846bdbaf5749b06922e8101cf6331d79e29a562816fd29640ec: Status 404 returned error can't find the container with id 26922dac3bbc7846bdbaf5749b06922e8101cf6331d79e29a562816fd29640ec Jul 10 17:26:41.763921 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.763889 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:26:41.764024 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.763970 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:26:41.764085 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:41.764019 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:26:42.666273 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:42.666238 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" event={"ID":"d9d4cd98-d80e-421c-b91e-7558bf2dae5a","Type":"ContainerStarted","Data":"8029b46481de6b739007dcdb792784f4b8b64748068107eb2f0173b6aa99efd2"} Jul 10 17:26:42.666803 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:42.666293 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" event={"ID":"d9d4cd98-d80e-421c-b91e-7558bf2dae5a","Type":"ContainerStarted","Data":"26922dac3bbc7846bdbaf5749b06922e8101cf6331d79e29a562816fd29640ec"} Jul 10 17:26:42.687395 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:42.687346 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" podStartSLOduration=1.687330218 podStartE2EDuration="1.687330218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:26:42.684874092 +0000 UTC m=+1942.810582505" watchObservedRunningTime="2026-07-10 17:26:42.687330218 +0000 UTC m=+1942.813038616" Jul 10 17:26:43.616229 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:43.616190 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:43.621139 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:43.621111 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:43.671992 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:43.671958 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:43.673039 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:43.673019 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:26:44.134988 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:44.134950 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:26:44.658526 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:44.658471 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:44.758865 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:44.758833 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:44.878299 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:44.878249 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:45.183381 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:45.183331 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:45.471775 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:45.471660 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:45.578792 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:45.578762 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:49.240363 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:49.240327 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:57.670792 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:57.670728 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:57.771210 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:57.771180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:58.428330 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:58.428292 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:26:59.713747 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:59.713705 2559 patch_prober.go:28] interesting pod/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 17:26:59.714141 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:26:59.713774 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" podUID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 17:27:00.765560 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:00.765457 2559 generic.go:356] "Generic (PLEG): container finished" podID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerID="8029b46481de6b739007dcdb792784f4b8b64748068107eb2f0173b6aa99efd2" exitCode=0 Jul 10 17:27:00.765560 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:00.765537 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" event={"ID":"d9d4cd98-d80e-421c-b91e-7558bf2dae5a","Type":"ContainerDied","Data":"8029b46481de6b739007dcdb792784f4b8b64748068107eb2f0173b6aa99efd2"} Jul 10 17:27:00.765986 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:00.765973 2559 scope.go:117] "RemoveContainer" containerID="8029b46481de6b739007dcdb792784f4b8b64748068107eb2f0173b6aa99efd2" Jul 10 17:27:00.767083 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:00.767057 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:27:00.767179 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:00.767120 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:27:00.767179 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:00.767152 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:27:01.772918 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:01.772878 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" event={"ID":"d9d4cd98-d80e-421c-b91e-7558bf2dae5a","Type":"ContainerStarted","Data":"91b3395e3c7ad157660d4814a82a1d22370a4d4844c98d4a91fb723433752634"} Jul 10 17:27:02.616360 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:02.616320 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:27:02.616615 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:02.616588 2559 patch_prober.go:28] interesting pod/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 10 17:27:02.616683 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:02.616660 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" podUID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 10 17:27:03.556336 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.556303 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5"] Jul 10 17:27:03.559050 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.559033 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.562904 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.562875 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cfd078\"/\"test-model-cfd078-kserve-self-signed-certs\"" Jul 10 17:27:03.563061 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.562939 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cfd078\"/\"kube-root-ca.crt\"" Jul 10 17:27:03.563061 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.562953 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cfd078\"/\"openshift-service-ca.crt\"" Jul 10 17:27:03.563061 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.562938 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cfd078\"/\"default-dockercfg-qb8qd\"" Jul 10 17:27:03.572146 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.572113 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5"] Jul 10 17:27:03.591527 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.585029 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:27:03.616761 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.616715 2559 patch_prober.go:28] interesting pod/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 10 17:27:03.616949 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.616777 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" podUID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 10 17:27:03.626884 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.626843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-home\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.627024 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.626892 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3230a1-b1a8-428b-986b-fd1059e26c8a-tls-certs\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.627024 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.626970 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvt6f\" (UniqueName: \"kubernetes.io/projected/eb3230a1-b1a8-428b-986b-fd1059e26c8a-kube-api-access-jvt6f\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.627024 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.627014 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-tmp-dir\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.627208 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.627093 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-model-cache\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.627208 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.627119 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-dshm\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.727984 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.727946 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvt6f\" (UniqueName: \"kubernetes.io/projected/eb3230a1-b1a8-428b-986b-fd1059e26c8a-kube-api-access-jvt6f\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.728176 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.727995 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-tmp-dir\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.728176 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.728036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-model-cache\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.728176 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.728054 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-dshm\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.728176 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.728118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-home\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.728176 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.728155 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3230a1-b1a8-428b-986b-fd1059e26c8a-tls-certs\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.728460 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.728429 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-tmp-dir\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.728557 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.728531 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-model-cache\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.728626 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.728608 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-home\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.730656 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.730627 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-dshm\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.731030 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.731009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3230a1-b1a8-428b-986b-fd1059e26c8a-tls-certs\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.735576 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.735555 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvt6f\" (UniqueName: \"kubernetes.io/projected/eb3230a1-b1a8-428b-986b-fd1059e26c8a-kube-api-access-jvt6f\") pod \"test-model-cfd078-kserve-5c4b869bc7-b68v5\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:03.869891 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:03.869830 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:04.060506 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:04.060468 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5"] Jul 10 17:27:04.061466 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:27:04.061440 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb3230a1_b1a8_428b_986b_fd1059e26c8a.slice/crio-b98ea7e191f94508247b1607844a9c17541636d1fc336c797e49c972ff1331d3 WatchSource:0}: Error finding container b98ea7e191f94508247b1607844a9c17541636d1fc336c797e49c972ff1331d3: Status 404 returned error can't find the container with id b98ea7e191f94508247b1607844a9c17541636d1fc336c797e49c972ff1331d3 Jul 10 17:27:04.615857 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:04.615824 2559 patch_prober.go:28] interesting pod/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 10 17:27:04.616251 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:04.615881 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" podUID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 10 17:27:04.797348 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:04.797293 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" event={"ID":"eb3230a1-b1a8-428b-986b-fd1059e26c8a","Type":"ContainerStarted","Data":"a9623d7c1779b53a45539c2486f0cc4856710d8e29d5ad49b24fe8d45a2bc333"} Jul 10 17:27:04.797585 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:04.797361 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" event={"ID":"eb3230a1-b1a8-428b-986b-fd1059e26c8a","Type":"ContainerStarted","Data":"b98ea7e191f94508247b1607844a9c17541636d1fc336c797e49c972ff1331d3"} Jul 10 17:27:04.815682 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:04.815617 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" podStartSLOduration=1.815599299 podStartE2EDuration="1.815599299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:27:04.813993162 +0000 UTC m=+1964.939701575" watchObservedRunningTime="2026-07-10 17:27:04.815599299 +0000 UTC m=+1964.941307787" Jul 10 17:27:05.616279 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:05.616234 2559 patch_prober.go:28] interesting pod/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 10 17:27:05.616723 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:05.616309 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" podUID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 10 17:27:06.143084 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:06.143046 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:27:06.262911 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:06.262875 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:27:06.364088 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:06.364057 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:27:06.620168 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:06.620131 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:27:06.620640 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:06.620442 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:27:06.621312 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:06.621291 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:27:13.870233 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:13.870136 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:13.870233 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:13.870190 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:13.882877 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:13.882835 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:14.860485 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:14.860453 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:27:19.827968 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:19.827933 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:27:30.660002 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:30.659961 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:27:35.842628 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:35.842592 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:27:48.456316 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:48.456275 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5"] Jul 10 17:27:48.474797 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:48.474760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:27:49.406321 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:49.406281 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:27:49.586091 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:49.586052 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:27:50.010562 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:50.010528 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:27:50.108759 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:50.108720 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:27:50.209369 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:50.209329 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:27:50.313075 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:50.312987 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:27:54.937300 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:54.937254 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:27:55.085401 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:27:55.085360 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:03.478993 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.478953 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" podUID="eb3230a1-b1a8-428b-986b-fd1059e26c8a" containerName="main" containerID="cri-o://a9623d7c1779b53a45539c2486f0cc4856710d8e29d5ad49b24fe8d45a2bc333" gracePeriod=45 Jul 10 17:28:03.729729 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.729669 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:28:03.788760 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.788726 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-model-cache\") pod \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " Jul 10 17:28:03.788760 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.788765 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-dshm\") pod \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " Jul 10 17:28:03.788990 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.788805 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvt6f\" (UniqueName: \"kubernetes.io/projected/eb3230a1-b1a8-428b-986b-fd1059e26c8a-kube-api-access-jvt6f\") pod \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " Jul 10 17:28:03.788990 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.788849 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-home\") pod \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " Jul 10 17:28:03.788990 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.788903 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3230a1-b1a8-428b-986b-fd1059e26c8a-tls-certs\") pod \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " Jul 10 17:28:03.789153 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.789005 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-tmp-dir\") pod \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\" (UID: \"eb3230a1-b1a8-428b-986b-fd1059e26c8a\") " Jul 10 17:28:03.789209 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.788997 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb3230a1-b1a8-428b-986b-fd1059e26c8a" (UID: "eb3230a1-b1a8-428b-986b-fd1059e26c8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:03.789209 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.789149 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-home" (OuterVolumeSpecName: "home") pod "eb3230a1-b1a8-428b-986b-fd1059e26c8a" (UID: "eb3230a1-b1a8-428b-986b-fd1059e26c8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:03.789299 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.789280 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb3230a1-b1a8-428b-986b-fd1059e26c8a" (UID: "eb3230a1-b1a8-428b-986b-fd1059e26c8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:03.789349 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.789331 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-model-cache\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:03.789391 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.789347 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-home\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:03.791235 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.791201 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb3230a1-b1a8-428b-986b-fd1059e26c8a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb3230a1-b1a8-428b-986b-fd1059e26c8a" (UID: "eb3230a1-b1a8-428b-986b-fd1059e26c8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:28:03.791352 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.791246 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-dshm" (OuterVolumeSpecName: "dshm") pod "eb3230a1-b1a8-428b-986b-fd1059e26c8a" (UID: "eb3230a1-b1a8-428b-986b-fd1059e26c8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:03.791352 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.791274 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3230a1-b1a8-428b-986b-fd1059e26c8a-kube-api-access-jvt6f" (OuterVolumeSpecName: "kube-api-access-jvt6f") pod "eb3230a1-b1a8-428b-986b-fd1059e26c8a" (UID: "eb3230a1-b1a8-428b-986b-fd1059e26c8a"). InnerVolumeSpecName "kube-api-access-jvt6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:28:03.890741 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.890704 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-tmp-dir\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:03.890741 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.890735 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3230a1-b1a8-428b-986b-fd1059e26c8a-dshm\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:03.890741 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.890747 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvt6f\" (UniqueName: \"kubernetes.io/projected/eb3230a1-b1a8-428b-986b-fd1059e26c8a-kube-api-access-jvt6f\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:03.891049 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:03.890757 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3230a1-b1a8-428b-986b-fd1059e26c8a-tls-certs\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:04.120175 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:04.120140 2559 generic.go:356] "Generic (PLEG): container finished" podID="eb3230a1-b1a8-428b-986b-fd1059e26c8a" containerID="a9623d7c1779b53a45539c2486f0cc4856710d8e29d5ad49b24fe8d45a2bc333" exitCode=0 Jul 10 17:28:04.120175 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:04.120178 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" Jul 10 17:28:04.120390 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:04.120211 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" event={"ID":"eb3230a1-b1a8-428b-986b-fd1059e26c8a","Type":"ContainerDied","Data":"a9623d7c1779b53a45539c2486f0cc4856710d8e29d5ad49b24fe8d45a2bc333"} Jul 10 17:28:04.120390 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:04.120248 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5" event={"ID":"eb3230a1-b1a8-428b-986b-fd1059e26c8a","Type":"ContainerDied","Data":"b98ea7e191f94508247b1607844a9c17541636d1fc336c797e49c972ff1331d3"} Jul 10 17:28:04.120390 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:04.120265 2559 scope.go:117] "RemoveContainer" containerID="a9623d7c1779b53a45539c2486f0cc4856710d8e29d5ad49b24fe8d45a2bc333" Jul 10 17:28:04.130384 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:04.130364 2559 scope.go:117] "RemoveContainer" containerID="a9623d7c1779b53a45539c2486f0cc4856710d8e29d5ad49b24fe8d45a2bc333" Jul 10 17:28:04.130695 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:28:04.130677 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9623d7c1779b53a45539c2486f0cc4856710d8e29d5ad49b24fe8d45a2bc333\": container with ID starting with a9623d7c1779b53a45539c2486f0cc4856710d8e29d5ad49b24fe8d45a2bc333 not found: ID does not exist" containerID="a9623d7c1779b53a45539c2486f0cc4856710d8e29d5ad49b24fe8d45a2bc333" Jul 10 17:28:04.130759 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:04.130705 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9623d7c1779b53a45539c2486f0cc4856710d8e29d5ad49b24fe8d45a2bc333"} err="failed to get container status \"a9623d7c1779b53a45539c2486f0cc4856710d8e29d5ad49b24fe8d45a2bc333\": rpc error: code = NotFound desc = could not find container \"a9623d7c1779b53a45539c2486f0cc4856710d8e29d5ad49b24fe8d45a2bc333\": container with ID starting with a9623d7c1779b53a45539c2486f0cc4856710d8e29d5ad49b24fe8d45a2bc333 not found: ID does not exist" Jul 10 17:28:04.141959 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:04.141923 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5"] Jul 10 17:28:04.144070 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:04.144050 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-cfd078/test-model-cfd078-kserve-5c4b869bc7-b68v5"] Jul 10 17:28:04.399510 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:04.399407 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb3230a1-b1a8-428b-986b-fd1059e26c8a" path="/var/lib/kubelet/pods/eb3230a1-b1a8-428b-986b-fd1059e26c8a/volumes" Jul 10 17:28:31.788796 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:31.788765 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:32.077381 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:32.077259 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:32.321468 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:32.321412 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:32.485211 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:32.485148 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:32.919823 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:32.919787 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:33.027303 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:33.027270 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:33.170141 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:33.170053 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:35.414724 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:35.414687 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:35.513393 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:35.513361 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:35.536595 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:35.536550 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:28:38.757200 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:38.757164 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:38.824750 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:38.824724 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:39.028579 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:39.028478 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4"] Jul 10 17:28:39.028781 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:39.028756 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" podUID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerName="istio-proxy" containerID="cri-o://91b3395e3c7ad157660d4814a82a1d22370a4d4844c98d4a91fb723433752634" gracePeriod=30 Jul 10 17:28:39.049604 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:39.049569 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:39.298348 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:39.298255 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:39.670743 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:39.670710 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:40.005824 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:40.005734 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:40.581179 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:40.581069 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:42.722133 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:42.722097 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:42.810000 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:42.809963 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:42.987222 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:42.987135 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:43.467782 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:43.467739 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:43.564868 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:43.564835 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:43.668388 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:43.668359 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:44.301886 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.301816 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:28:44.346017 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.345980 2559 generic.go:356] "Generic (PLEG): container finished" podID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerID="91b3395e3c7ad157660d4814a82a1d22370a4d4844c98d4a91fb723433752634" exitCode=0 Jul 10 17:28:44.346226 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.346026 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" event={"ID":"d9d4cd98-d80e-421c-b91e-7558bf2dae5a","Type":"ContainerDied","Data":"91b3395e3c7ad157660d4814a82a1d22370a4d4844c98d4a91fb723433752634"} Jul 10 17:28:44.346226 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.346049 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" Jul 10 17:28:44.346226 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.346102 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4" event={"ID":"d9d4cd98-d80e-421c-b91e-7558bf2dae5a","Type":"ContainerDied","Data":"26922dac3bbc7846bdbaf5749b06922e8101cf6331d79e29a562816fd29640ec"} Jul 10 17:28:44.346226 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.346135 2559 scope.go:117] "RemoveContainer" containerID="91b3395e3c7ad157660d4814a82a1d22370a4d4844c98d4a91fb723433752634" Jul 10 17:28:44.355617 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.355598 2559 scope.go:117] "RemoveContainer" containerID="8029b46481de6b739007dcdb792784f4b8b64748068107eb2f0173b6aa99efd2" Jul 10 17:28:44.359889 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.359867 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-data\") pod \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " Jul 10 17:28:44.359981 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.359906 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-podinfo\") pod \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " Jul 10 17:28:44.359981 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.359963 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istiod-ca-cert\") pod \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " Jul 10 17:28:44.360098 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.359986 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p95vd\" (UniqueName: \"kubernetes.io/projected/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-kube-api-access-p95vd\") pod \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " Jul 10 17:28:44.360098 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.360036 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-workload-certs\") pod \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " Jul 10 17:28:44.360206 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.360113 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-credential-socket\") pod \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " Jul 10 17:28:44.360206 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.360136 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-token\") pod \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " Jul 10 17:28:44.360206 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.360160 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-workload-socket\") pod \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " Jul 10 17:28:44.360206 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.360185 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-envoy\") pod \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\" (UID: \"d9d4cd98-d80e-421c-b91e-7558bf2dae5a\") " Jul 10 17:28:44.360443 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.360403 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9d4cd98-d80e-421c-b91e-7558bf2dae5a" (UID: "d9d4cd98-d80e-421c-b91e-7558bf2dae5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:44.360561 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.360516 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-credential-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:44.360694 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.360672 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9d4cd98-d80e-421c-b91e-7558bf2dae5a" (UID: "d9d4cd98-d80e-421c-b91e-7558bf2dae5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:44.360768 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.360683 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9d4cd98-d80e-421c-b91e-7558bf2dae5a" (UID: "d9d4cd98-d80e-421c-b91e-7558bf2dae5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:44.360768 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.360699 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9d4cd98-d80e-421c-b91e-7558bf2dae5a" (UID: "d9d4cd98-d80e-421c-b91e-7558bf2dae5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:28:44.360948 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.360926 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9d4cd98-d80e-421c-b91e-7558bf2dae5a" (UID: "d9d4cd98-d80e-421c-b91e-7558bf2dae5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:44.362716 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.362678 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9d4cd98-d80e-421c-b91e-7558bf2dae5a" (UID: "d9d4cd98-d80e-421c-b91e-7558bf2dae5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:28:44.362844 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.362823 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9d4cd98-d80e-421c-b91e-7558bf2dae5a" (UID: "d9d4cd98-d80e-421c-b91e-7558bf2dae5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:44.363070 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.363052 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9d4cd98-d80e-421c-b91e-7558bf2dae5a" (UID: "d9d4cd98-d80e-421c-b91e-7558bf2dae5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:28:44.363143 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.363114 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-kube-api-access-p95vd" (OuterVolumeSpecName: "kube-api-access-p95vd") pod "d9d4cd98-d80e-421c-b91e-7558bf2dae5a" (UID: "d9d4cd98-d80e-421c-b91e-7558bf2dae5a"). InnerVolumeSpecName "kube-api-access-p95vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:28:44.365585 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.365563 2559 scope.go:117] "RemoveContainer" containerID="91b3395e3c7ad157660d4814a82a1d22370a4d4844c98d4a91fb723433752634" Jul 10 17:28:44.365879 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:28:44.365860 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b3395e3c7ad157660d4814a82a1d22370a4d4844c98d4a91fb723433752634\": container with ID starting with 91b3395e3c7ad157660d4814a82a1d22370a4d4844c98d4a91fb723433752634 not found: ID does not exist" containerID="91b3395e3c7ad157660d4814a82a1d22370a4d4844c98d4a91fb723433752634" Jul 10 17:28:44.365963 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.365891 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b3395e3c7ad157660d4814a82a1d22370a4d4844c98d4a91fb723433752634"} err="failed to get container status \"91b3395e3c7ad157660d4814a82a1d22370a4d4844c98d4a91fb723433752634\": rpc error: code = NotFound desc = could not find container \"91b3395e3c7ad157660d4814a82a1d22370a4d4844c98d4a91fb723433752634\": container with ID starting with 91b3395e3c7ad157660d4814a82a1d22370a4d4844c98d4a91fb723433752634 not found: ID does not exist" Jul 10 17:28:44.365963 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.365916 2559 scope.go:117] "RemoveContainer" containerID="8029b46481de6b739007dcdb792784f4b8b64748068107eb2f0173b6aa99efd2" Jul 10 17:28:44.366229 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:28:44.366211 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8029b46481de6b739007dcdb792784f4b8b64748068107eb2f0173b6aa99efd2\": container with ID starting with 8029b46481de6b739007dcdb792784f4b8b64748068107eb2f0173b6aa99efd2 not found: ID does not exist" containerID="8029b46481de6b739007dcdb792784f4b8b64748068107eb2f0173b6aa99efd2" Jul 10 17:28:44.366280 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.366235 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8029b46481de6b739007dcdb792784f4b8b64748068107eb2f0173b6aa99efd2"} err="failed to get container status \"8029b46481de6b739007dcdb792784f4b8b64748068107eb2f0173b6aa99efd2\": rpc error: code = NotFound desc = could not find container \"8029b46481de6b739007dcdb792784f4b8b64748068107eb2f0173b6aa99efd2\": container with ID starting with 8029b46481de6b739007dcdb792784f4b8b64748068107eb2f0173b6aa99efd2 not found: ID does not exist" Jul 10 17:28:44.461695 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.461650 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-workload-certs\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:44.461695 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.461691 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-token\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:44.461695 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.461703 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-workload-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:44.461956 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.461716 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-envoy\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:44.461956 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.461730 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-data\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:44.461956 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.461742 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istio-podinfo\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:44.461956 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.461755 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-istiod-ca-cert\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:44.461956 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.461768 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p95vd\" (UniqueName: \"kubernetes.io/projected/d9d4cd98-d80e-421c-b91e-7558bf2dae5a-kube-api-access-p95vd\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:28:44.663965 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.663934 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4"] Jul 10 17:28:44.667652 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:44.667620 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fc13d1-openshift-default-76d76cb75f-jmdl4"] Jul 10 17:28:46.399459 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:46.399418 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" path="/var/lib/kubelet/pods/d9d4cd98-d80e-421c-b91e-7558bf2dae5a/volumes" Jul 10 17:28:47.261882 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:47.261849 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:47.780001 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:47.779961 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:48.086886 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.086788 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz"] Jul 10 17:28:48.087366 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.087349 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerName="istio-proxy" Jul 10 17:28:48.087469 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.087367 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerName="istio-proxy" Jul 10 17:28:48.087469 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.087394 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerName="istio-proxy" Jul 10 17:28:48.087469 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.087404 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerName="istio-proxy" Jul 10 17:28:48.087469 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.087436 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3230a1-b1a8-428b-986b-fd1059e26c8a" containerName="main" Jul 10 17:28:48.087469 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.087445 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3230a1-b1a8-428b-986b-fd1059e26c8a" containerName="main" Jul 10 17:28:48.087734 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.087539 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerName="istio-proxy" Jul 10 17:28:48.087734 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.087552 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb3230a1-b1a8-428b-986b-fd1059e26c8a" containerName="main" Jul 10 17:28:48.087734 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.087564 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d4cd98-d80e-421c-b91e-7558bf2dae5a" containerName="istio-proxy" Jul 10 17:28:48.092416 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.092394 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.095207 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.095183 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ad6ba3-openshift-default-dockercfg-mgghl\"" Jul 10 17:28:48.100780 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.100747 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz"] Jul 10 17:28:48.131079 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.131048 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:48.196657 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.196617 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-workload-socket\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.196870 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.196665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxdp\" (UniqueName: \"kubernetes.io/projected/a4331d1a-0347-43f4-9c3b-59a71a474212-kube-api-access-zdxdp\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.196870 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.196704 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-data\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.196870 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.196757 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-workload-certs\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.196870 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.196783 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-envoy\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.196870 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.196812 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-credential-socket\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.196870 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.196858 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-podinfo\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.197091 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.196938 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-token\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.197091 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.196962 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4331d1a-0347-43f4-9c3b-59a71a474212-istiod-ca-cert\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.237368 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.237328 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:48.298149 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.298118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-workload-socket\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.298149 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.298152 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxdp\" (UniqueName: \"kubernetes.io/projected/a4331d1a-0347-43f4-9c3b-59a71a474212-kube-api-access-zdxdp\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.298411 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.298173 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-data\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.298411 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.298200 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-workload-certs\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.298411 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.298215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-envoy\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.298411 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.298235 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-credential-socket\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.298411 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.298256 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-podinfo\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.298411 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.298321 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-token\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.298411 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.298347 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4331d1a-0347-43f4-9c3b-59a71a474212-istiod-ca-cert\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.298727 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.298548 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-workload-socket\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.298784 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.298758 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-data\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.298846 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.298797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-credential-socket\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.298896 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.298835 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-workload-certs\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.299151 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.299134 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4331d1a-0347-43f4-9c3b-59a71a474212-istiod-ca-cert\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.300771 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.300746 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-envoy\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.300981 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.300962 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-podinfo\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.306581 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.306559 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-token\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.306762 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.306739 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxdp\" (UniqueName: \"kubernetes.io/projected/a4331d1a-0347-43f4-9c3b-59a71a474212-kube-api-access-zdxdp\") pod \"e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.404389 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.404357 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:48.546503 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.546461 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz"] Jul 10 17:28:48.547386 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:28:48.547358 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4331d1a_0347_43f4_9c3b_59a71a474212.slice/crio-238b0c92a23797d23ee233fbdaeb0f97a0c41bcaa27165fc0099c8b8b02cbaba WatchSource:0}: Error finding container 238b0c92a23797d23ee233fbdaeb0f97a0c41bcaa27165fc0099c8b8b02cbaba: Status 404 returned error can't find the container with id 238b0c92a23797d23ee233fbdaeb0f97a0c41bcaa27165fc0099c8b8b02cbaba Jul 10 17:28:48.549991 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.549956 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:28:48.550072 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.550042 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:28:48.550111 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:48.550093 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:28:49.383434 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:49.383393 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" event={"ID":"a4331d1a-0347-43f4-9c3b-59a71a474212","Type":"ContainerStarted","Data":"568af4afa1d91b15b8e55616fbb6e252675452c8ab5b0d31fdbcd5b66e20a07c"} Jul 10 17:28:49.383958 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:49.383483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" event={"ID":"a4331d1a-0347-43f4-9c3b-59a71a474212","Type":"ContainerStarted","Data":"238b0c92a23797d23ee233fbdaeb0f97a0c41bcaa27165fc0099c8b8b02cbaba"} Jul 10 17:28:49.405378 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:49.405337 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:49.405675 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:49.405641 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" podStartSLOduration=1.405629008 podStartE2EDuration="1.405629008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:28:49.403749561 +0000 UTC m=+2069.529457970" watchObservedRunningTime="2026-07-10 17:28:49.405629008 +0000 UTC m=+2069.531337429" Jul 10 17:28:49.410059 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:49.410035 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:50.389638 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:50.389608 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:50.390558 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:50.390531 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:28:50.908782 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:50.908743 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:28:51.490602 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:51.490554 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:51.569180 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:51.569145 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:51.642442 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:51.642408 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:51.872806 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:51.872775 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:52.051782 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:52.051749 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:52.153088 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:52.152996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:28:55.982823 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:28:55.982794 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:04.364973 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:04.364937 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:04.463154 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:04.463109 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:04.563256 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:04.563221 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:04.655384 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:04.655280 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:04.930769 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:04.930686 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:06.084357 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:06.084312 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:06.431779 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:06.431736 2559 patch_prober.go:28] interesting pod/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 17:29:06.431968 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:06.431812 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" podUID="a4331d1a-0347-43f4-9c3b-59a71a474212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 17:29:08.494469 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:08.494436 2559 generic.go:356] "Generic (PLEG): container finished" podID="a4331d1a-0347-43f4-9c3b-59a71a474212" containerID="568af4afa1d91b15b8e55616fbb6e252675452c8ab5b0d31fdbcd5b66e20a07c" exitCode=0 Jul 10 17:29:08.494938 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:08.494483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" event={"ID":"a4331d1a-0347-43f4-9c3b-59a71a474212","Type":"ContainerDied","Data":"568af4afa1d91b15b8e55616fbb6e252675452c8ab5b0d31fdbcd5b66e20a07c"} Jul 10 17:29:08.494938 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:08.494872 2559 scope.go:117] "RemoveContainer" containerID="568af4afa1d91b15b8e55616fbb6e252675452c8ab5b0d31fdbcd5b66e20a07c" Jul 10 17:29:08.496010 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:08.495977 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:29:08.496088 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:08.496059 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:29:08.496130 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:08.496096 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:29:08.635333 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:08.635294 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:29:09.508252 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:09.508201 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" event={"ID":"a4331d1a-0347-43f4-9c3b-59a71a474212","Type":"ContainerStarted","Data":"e5ee9a2fc5da1bb9aca242513738fc351d91f202de9686bf63353880391b9932"} Jul 10 17:29:10.404591 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:10.404548 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:29:10.404855 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:10.404825 2559 patch_prober.go:28] interesting pod/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 10 17:29:10.404949 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:10.404923 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" podUID="a4331d1a-0347-43f4-9c3b-59a71a474212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 10 17:29:11.025511 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:11.025461 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:11.116259 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:11.116225 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:11.338758 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:11.338735 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:11.405169 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:11.405127 2559 patch_prober.go:28] interesting pod/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 10 17:29:11.405518 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:11.405458 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" podUID="a4331d1a-0347-43f4-9c3b-59a71a474212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 10 17:29:11.616145 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:11.616054 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:11.772615 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:11.772583 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:12.238637 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:12.238604 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:12.383290 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:12.383256 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:12.404982 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:12.404951 2559 patch_prober.go:28] interesting pod/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 10 17:29:12.405266 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:12.405234 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" podUID="a4331d1a-0347-43f4-9c3b-59a71a474212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 10 17:29:12.490367 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:12.490263 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:12.690979 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:12.690945 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:13.415555 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:13.415516 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:29:13.416037 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:13.415915 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:29:13.416969 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:13.416937 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:29:17.846133 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:17.846098 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:17.945015 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:17.944980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:18.096356 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:18.096283 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz"] Jul 10 17:29:18.096542 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:18.096510 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" podUID="a4331d1a-0347-43f4-9c3b-59a71a474212" containerName="istio-proxy" containerID="cri-o://e5ee9a2fc5da1bb9aca242513738fc351d91f202de9686bf63353880391b9932" gracePeriod=30 Jul 10 17:29:18.122897 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:18.122862 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:18.407981 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:18.407875 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:18.782220 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:18.782113 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:19.120802 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:19.120771 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:19.528530 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:19.528400 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:21.941822 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:21.941783 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:22.030544 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:22.030510 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:22.098293 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:22.098261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:22.198402 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:22.198286 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:22.336512 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:22.336452 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:29:22.616645 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:22.616614 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:22.716535 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:22.716482 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:23.380675 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.380646 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:29:23.446243 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446207 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-workload-socket\") pod \"a4331d1a-0347-43f4-9c3b-59a71a474212\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " Jul 10 17:29:23.446443 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446253 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4331d1a-0347-43f4-9c3b-59a71a474212-istiod-ca-cert\") pod \"a4331d1a-0347-43f4-9c3b-59a71a474212\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " Jul 10 17:29:23.446443 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446291 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-workload-certs\") pod \"a4331d1a-0347-43f4-9c3b-59a71a474212\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " Jul 10 17:29:23.446443 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446323 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-envoy\") pod \"a4331d1a-0347-43f4-9c3b-59a71a474212\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " Jul 10 17:29:23.446443 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446352 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdxdp\" (UniqueName: \"kubernetes.io/projected/a4331d1a-0347-43f4-9c3b-59a71a474212-kube-api-access-zdxdp\") pod \"a4331d1a-0347-43f4-9c3b-59a71a474212\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " Jul 10 17:29:23.446443 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446422 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-data\") pod \"a4331d1a-0347-43f4-9c3b-59a71a474212\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " Jul 10 17:29:23.446750 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446452 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-podinfo\") pod \"a4331d1a-0347-43f4-9c3b-59a71a474212\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " Jul 10 17:29:23.446750 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446477 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-token\") pod \"a4331d1a-0347-43f4-9c3b-59a71a474212\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " Jul 10 17:29:23.446750 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446540 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-credential-socket\") pod \"a4331d1a-0347-43f4-9c3b-59a71a474212\" (UID: \"a4331d1a-0347-43f4-9c3b-59a71a474212\") " Jul 10 17:29:23.446750 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446597 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a4331d1a-0347-43f4-9c3b-59a71a474212" (UID: "a4331d1a-0347-43f4-9c3b-59a71a474212"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:29:23.446750 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446638 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a4331d1a-0347-43f4-9c3b-59a71a474212" (UID: "a4331d1a-0347-43f4-9c3b-59a71a474212"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:29:23.447001 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446844 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a4331d1a-0347-43f4-9c3b-59a71a474212" (UID: "a4331d1a-0347-43f4-9c3b-59a71a474212"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:29:23.447001 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446887 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-workload-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:29:23.447001 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.446906 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-workload-certs\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:29:23.447185 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.447071 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4331d1a-0347-43f4-9c3b-59a71a474212-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a4331d1a-0347-43f4-9c3b-59a71a474212" (UID: "a4331d1a-0347-43f4-9c3b-59a71a474212"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:29:23.447643 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.447595 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-data" (OuterVolumeSpecName: "istio-data") pod "a4331d1a-0347-43f4-9c3b-59a71a474212" (UID: "a4331d1a-0347-43f4-9c3b-59a71a474212"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:29:23.448845 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.448812 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-token" (OuterVolumeSpecName: "istio-token") pod "a4331d1a-0347-43f4-9c3b-59a71a474212" (UID: "a4331d1a-0347-43f4-9c3b-59a71a474212"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:29:23.448953 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.448875 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a4331d1a-0347-43f4-9c3b-59a71a474212" (UID: "a4331d1a-0347-43f4-9c3b-59a71a474212"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:29:23.448953 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.448886 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4331d1a-0347-43f4-9c3b-59a71a474212-kube-api-access-zdxdp" (OuterVolumeSpecName: "kube-api-access-zdxdp") pod "a4331d1a-0347-43f4-9c3b-59a71a474212" (UID: "a4331d1a-0347-43f4-9c3b-59a71a474212"). InnerVolumeSpecName "kube-api-access-zdxdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:29:23.448953 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.448915 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a4331d1a-0347-43f4-9c3b-59a71a474212" (UID: "a4331d1a-0347-43f4-9c3b-59a71a474212"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:29:23.547466 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.547419 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4331d1a-0347-43f4-9c3b-59a71a474212-istiod-ca-cert\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:29:23.547466 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.547460 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-envoy\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:29:23.547466 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.547470 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdxdp\" (UniqueName: \"kubernetes.io/projected/a4331d1a-0347-43f4-9c3b-59a71a474212-kube-api-access-zdxdp\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:29:23.547466 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.547479 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-data\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:29:23.547466 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.547487 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-podinfo\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:29:23.547811 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.547517 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4331d1a-0347-43f4-9c3b-59a71a474212-istio-token\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:29:23.547811 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.547526 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4331d1a-0347-43f4-9c3b-59a71a474212-credential-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:29:23.600161 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.600125 2559 generic.go:356] "Generic (PLEG): container finished" podID="a4331d1a-0347-43f4-9c3b-59a71a474212" containerID="e5ee9a2fc5da1bb9aca242513738fc351d91f202de9686bf63353880391b9932" exitCode=0 Jul 10 17:29:23.600319 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.600191 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" Jul 10 17:29:23.600319 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.600213 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" event={"ID":"a4331d1a-0347-43f4-9c3b-59a71a474212","Type":"ContainerDied","Data":"e5ee9a2fc5da1bb9aca242513738fc351d91f202de9686bf63353880391b9932"} Jul 10 17:29:23.600319 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.600282 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz" event={"ID":"a4331d1a-0347-43f4-9c3b-59a71a474212","Type":"ContainerDied","Data":"238b0c92a23797d23ee233fbdaeb0f97a0c41bcaa27165fc0099c8b8b02cbaba"} Jul 10 17:29:23.600420 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.600316 2559 scope.go:117] "RemoveContainer" containerID="e5ee9a2fc5da1bb9aca242513738fc351d91f202de9686bf63353880391b9932" Jul 10 17:29:23.609955 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.609931 2559 scope.go:117] "RemoveContainer" containerID="568af4afa1d91b15b8e55616fbb6e252675452c8ab5b0d31fdbcd5b66e20a07c" Jul 10 17:29:23.618677 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.618660 2559 scope.go:117] "RemoveContainer" containerID="e5ee9a2fc5da1bb9aca242513738fc351d91f202de9686bf63353880391b9932" Jul 10 17:29:23.618964 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:29:23.618947 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ee9a2fc5da1bb9aca242513738fc351d91f202de9686bf63353880391b9932\": container with ID starting with e5ee9a2fc5da1bb9aca242513738fc351d91f202de9686bf63353880391b9932 not found: ID does not exist" containerID="e5ee9a2fc5da1bb9aca242513738fc351d91f202de9686bf63353880391b9932" Jul 10 17:29:23.619010 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.618972 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ee9a2fc5da1bb9aca242513738fc351d91f202de9686bf63353880391b9932"} err="failed to get container status \"e5ee9a2fc5da1bb9aca242513738fc351d91f202de9686bf63353880391b9932\": rpc error: code = NotFound desc = could not find container \"e5ee9a2fc5da1bb9aca242513738fc351d91f202de9686bf63353880391b9932\": container with ID starting with e5ee9a2fc5da1bb9aca242513738fc351d91f202de9686bf63353880391b9932 not found: ID does not exist" Jul 10 17:29:23.619010 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.618989 2559 scope.go:117] "RemoveContainer" containerID="568af4afa1d91b15b8e55616fbb6e252675452c8ab5b0d31fdbcd5b66e20a07c" Jul 10 17:29:23.619248 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:29:23.619229 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568af4afa1d91b15b8e55616fbb6e252675452c8ab5b0d31fdbcd5b66e20a07c\": container with ID starting with 568af4afa1d91b15b8e55616fbb6e252675452c8ab5b0d31fdbcd5b66e20a07c not found: ID does not exist" containerID="568af4afa1d91b15b8e55616fbb6e252675452c8ab5b0d31fdbcd5b66e20a07c" Jul 10 17:29:23.619300 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.619254 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568af4afa1d91b15b8e55616fbb6e252675452c8ab5b0d31fdbcd5b66e20a07c"} err="failed to get container status \"568af4afa1d91b15b8e55616fbb6e252675452c8ab5b0d31fdbcd5b66e20a07c\": rpc error: code = NotFound desc = could not find container \"568af4afa1d91b15b8e55616fbb6e252675452c8ab5b0d31fdbcd5b66e20a07c\": container with ID starting with 568af4afa1d91b15b8e55616fbb6e252675452c8ab5b0d31fdbcd5b66e20a07c not found: ID does not exist" Jul 10 17:29:23.622811 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.622786 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz"] Jul 10 17:29:23.627061 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:23.627031 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ad6ba3-openshift-default-f57cc844c-fkdmz"] Jul 10 17:29:24.399399 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:24.399361 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4331d1a-0347-43f4-9c3b-59a71a474212" path="/var/lib/kubelet/pods/a4331d1a-0347-43f4-9c3b-59a71a474212/volumes" Jul 10 17:29:26.521210 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:26.521176 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:26.884019 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:26.883984 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:27.180159 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.180071 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk"] Jul 10 17:29:27.180715 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.180694 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4331d1a-0347-43f4-9c3b-59a71a474212" containerName="istio-proxy" Jul 10 17:29:27.180715 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.180717 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4331d1a-0347-43f4-9c3b-59a71a474212" containerName="istio-proxy" Jul 10 17:29:27.180900 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.180732 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4331d1a-0347-43f4-9c3b-59a71a474212" containerName="istio-proxy" Jul 10 17:29:27.180900 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.180740 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4331d1a-0347-43f4-9c3b-59a71a474212" containerName="istio-proxy" Jul 10 17:29:27.180900 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.180848 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4331d1a-0347-43f4-9c3b-59a71a474212" containerName="istio-proxy" Jul 10 17:29:27.185575 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.185554 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.188140 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.188118 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-2482f8-openshift-default-dockercfg-x5mh9\"" Jul 10 17:29:27.198451 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.198424 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk"] Jul 10 17:29:27.212649 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.212616 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:27.284442 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.284396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-envoy\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.284665 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.284468 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-workload-certs\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.284665 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.284558 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-data\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.284665 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.284585 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-token\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.284797 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.284669 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-credential-socket\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.284797 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.284706 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istiod-ca-cert\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.284797 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.284756 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-workload-socket\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.284929 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.284794 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-podinfo\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.284929 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.284819 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwvt9\" (UniqueName: \"kubernetes.io/projected/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-kube-api-access-jwvt9\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.312466 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.312431 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:27.385618 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.385582 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-credential-socket\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.385799 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.385625 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istiod-ca-cert\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.385799 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.385660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-workload-socket\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.385799 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.385697 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-podinfo\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.385799 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.385725 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwvt9\" (UniqueName: \"kubernetes.io/projected/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-kube-api-access-jwvt9\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.385799 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.385783 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-envoy\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.386086 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.385865 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-workload-certs\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.386086 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.385934 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-data\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.386086 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.385966 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-token\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.386086 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.386052 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-credential-socket\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.386388 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.386363 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-workload-certs\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.386446 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.386399 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-data\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.386674 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.386657 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-workload-socket\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.386924 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.386903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istiod-ca-cert\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.388185 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.388161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-envoy\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.388396 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.388377 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-podinfo\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.394122 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.394099 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-token\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.394560 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.394544 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwvt9\" (UniqueName: \"kubernetes.io/projected/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-kube-api-access-jwvt9\") pod \"e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.496617 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.496531 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:27.637928 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.637895 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk"] Jul 10 17:29:27.638976 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:29:27.638950 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ef38a8_e30c_434d_9f8a_7c2ddbbf1bb2.slice/crio-21fdf3522b089991572d8b653c27cb57fb318ffb6eddd457d3cef24396f46b1d WatchSource:0}: Error finding container 21fdf3522b089991572d8b653c27cb57fb318ffb6eddd457d3cef24396f46b1d: Status 404 returned error can't find the container with id 21fdf3522b089991572d8b653c27cb57fb318ffb6eddd457d3cef24396f46b1d Jul 10 17:29:27.641232 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.641190 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:29:27.641307 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.641282 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:29:27.641345 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:27.641329 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:29:28.631028 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:28.630990 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" event={"ID":"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2","Type":"ContainerStarted","Data":"007a0ee32f638e239d9e359f5182bc77097ce8e9eb2f387f0a19ae637452b73d"} Jul 10 17:29:28.631198 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:28.631046 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" event={"ID":"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2","Type":"ContainerStarted","Data":"21fdf3522b089991572d8b653c27cb57fb318ffb6eddd457d3cef24396f46b1d"} Jul 10 17:29:28.652739 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:28.652679 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" podStartSLOduration=1.652661686 podStartE2EDuration="1.652661686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:29:28.650123935 +0000 UTC m=+2108.775832391" watchObservedRunningTime="2026-07-10 17:29:28.652661686 +0000 UTC m=+2108.778370095" Jul 10 17:29:29.497173 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:29.497135 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:29.502040 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:29.502016 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:29.636284 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:29.636252 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:29.637326 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:29.637302 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:30.597146 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:30.597109 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:30.695696 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:30.695662 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:30.871250 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:30.871170 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:31.416546 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:31.416478 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:31.515549 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:31.515512 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:31.616172 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:31.616140 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:35.173815 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:35.173784 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:38.238216 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:38.238173 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:29:38.714446 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:38.714415 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:43.431299 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:43.431261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:43.524317 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:43.524285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:45.677668 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:45.677630 2559 patch_prober.go:28] interesting pod/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 17:29:45.678074 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:45.677700 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" podUID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 17:29:46.740701 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:46.740607 2559 generic.go:356] "Generic (PLEG): container finished" podID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerID="007a0ee32f638e239d9e359f5182bc77097ce8e9eb2f387f0a19ae637452b73d" exitCode=0 Jul 10 17:29:46.740701 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:46.740688 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" event={"ID":"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2","Type":"ContainerDied","Data":"007a0ee32f638e239d9e359f5182bc77097ce8e9eb2f387f0a19ae637452b73d"} Jul 10 17:29:46.741133 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:46.741087 2559 scope.go:117] "RemoveContainer" containerID="007a0ee32f638e239d9e359f5182bc77097ce8e9eb2f387f0a19ae637452b73d" Jul 10 17:29:46.742216 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:46.742184 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:29:46.742295 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:46.742276 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:29:46.742342 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:46.742327 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 17:29:47.748557 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:47.748519 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" event={"ID":"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2","Type":"ContainerStarted","Data":"c70703aff4759537e94d75e05eef432323cacf8813b00d117032aed45ace311e"} Jul 10 17:29:48.497595 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:48.497555 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:48.497914 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:48.497883 2559 patch_prober.go:28] interesting pod/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 10 17:29:48.498140 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:48.498109 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" podUID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 10 17:29:49.497289 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:49.497248 2559 patch_prober.go:28] interesting pod/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 10 17:29:49.497931 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:49.497894 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" podUID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 10 17:29:50.497041 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:50.496992 2559 patch_prober.go:28] interesting pod/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 10 17:29:50.497260 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:50.497094 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" podUID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 10 17:29:51.497217 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:51.497168 2559 patch_prober.go:28] interesting pod/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 10 17:29:51.497780 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:51.497258 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" podUID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 10 17:29:52.500860 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:52.500822 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:52.501238 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:52.501106 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:52.501928 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:52.501911 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:29:53.323354 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:53.323321 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:53.420635 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:29:53.420602 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:30:02.041399 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.041349 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh"] Jul 10 17:30:02.042103 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.042085 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4331d1a-0347-43f4-9c3b-59a71a474212" containerName="istio-proxy" Jul 10 17:30:02.045091 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.045072 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.047995 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.047972 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-43091a\"/\"openshift-service-ca.crt\"" Jul 10 17:30:02.049467 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.049446 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-43091a\"/\"kube-root-ca.crt\"" Jul 10 17:30:02.049670 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.049650 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-43091a\"/\"auth-test-model-43091a-kserve-self-signed-certs\"" Jul 10 17:30:02.049862 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.049639 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-43091a\"/\"default-dockercfg-644pv\"" Jul 10 17:30:02.053183 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.053158 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh"] Jul 10 17:30:02.086867 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.086833 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8468276-c4a9-4170-a668-c463981e5b52-tls-certs\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.086867 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.086868 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-tmp-dir\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.087108 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.086890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-dshm\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.087108 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.086981 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvnm2\" (UniqueName: \"kubernetes.io/projected/d8468276-c4a9-4170-a668-c463981e5b52-kube-api-access-cvnm2\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.087108 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.087023 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-model-cache\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.087108 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.087065 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-home\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.183257 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.183215 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:30:02.188370 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.188340 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8468276-c4a9-4170-a668-c463981e5b52-tls-certs\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.188550 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.188385 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-tmp-dir\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.188550 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.188417 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-dshm\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.188550 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.188477 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnm2\" (UniqueName: \"kubernetes.io/projected/d8468276-c4a9-4170-a668-c463981e5b52-kube-api-access-cvnm2\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.188550 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.188544 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-model-cache\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.188754 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.188590 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-home\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.188801 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.188776 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-tmp-dir\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.188955 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.188933 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-model-cache\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.189029 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.189003 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-home\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.190882 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.190855 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-dshm\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.191169 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.191152 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8468276-c4a9-4170-a668-c463981e5b52-tls-certs\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.196916 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.196891 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvnm2\" (UniqueName: \"kubernetes.io/projected/d8468276-c4a9-4170-a668-c463981e5b52-kube-api-access-cvnm2\") pod \"auth-test-model-43091a-kserve-ff58b45f9-s4qkh\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.356913 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.356876 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:02.500778 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.500752 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh"] Jul 10 17:30:02.502286 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:30:02.502257 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8468276_c4a9_4170_a668_c463981e5b52.slice/crio-c93f3a162df2cc48c866664be7f664e2a778fc850ec7a39bf25eb426f11bd284 WatchSource:0}: Error finding container c93f3a162df2cc48c866664be7f664e2a778fc850ec7a39bf25eb426f11bd284: Status 404 returned error can't find the container with id c93f3a162df2cc48c866664be7f664e2a778fc850ec7a39bf25eb426f11bd284 Jul 10 17:30:02.837267 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.837171 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" event={"ID":"d8468276-c4a9-4170-a668-c463981e5b52","Type":"ContainerStarted","Data":"84d59b5416c1645705ec5ec5cdf3c624b4a10282a6bb3cfbc40ab188644f1562"} Jul 10 17:30:02.837267 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.837216 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" event={"ID":"d8468276-c4a9-4170-a668-c463981e5b52","Type":"ContainerStarted","Data":"c93f3a162df2cc48c866664be7f664e2a778fc850ec7a39bf25eb426f11bd284"} Jul 10 17:30:02.855991 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:02.855939 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" podStartSLOduration=0.855922971 podStartE2EDuration="855.922971ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:30:02.854646549 +0000 UTC m=+2142.980354970" watchObservedRunningTime="2026-07-10 17:30:02.855922971 +0000 UTC m=+2142.981631386" Jul 10 17:30:05.440050 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:05.440015 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:30:12.358134 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:12.358032 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:12.358134 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:12.358109 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:12.370810 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:12.370777 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:12.909303 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:12.909266 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:30:25.607234 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:25.607194 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:30:25.670731 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:25.670680 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:30:30.977442 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:30.977403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:30:34.439662 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:34.439618 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:30:49.238642 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:30:49.238600 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:31:09.136575 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:31:09.136535 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:31:26.937353 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:31:26.937312 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:31:44.148847 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:31:44.148760 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:32:01.736413 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:01.736374 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:32:10.782911 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:10.782870 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:32:18.181443 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:18.181410 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:32:19.537562 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:19.537519 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:32:33.832717 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:33.832679 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:32:37.387205 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:37.386921 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh"] Jul 10 17:32:37.399596 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:37.399564 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:32:38.347130 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:38.347090 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:32:44.057396 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:44.057358 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:32:51.033783 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:51.033745 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:32:52.413812 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.413767 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" podUID="d8468276-c4a9-4170-a668-c463981e5b52" containerName="main" containerID="cri-o://84d59b5416c1645705ec5ec5cdf3c624b4a10282a6bb3cfbc40ab188644f1562" gracePeriod=45 Jul 10 17:32:52.674454 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.674370 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:32:52.785021 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.784980 2559 generic.go:356] "Generic (PLEG): container finished" podID="d8468276-c4a9-4170-a668-c463981e5b52" containerID="84d59b5416c1645705ec5ec5cdf3c624b4a10282a6bb3cfbc40ab188644f1562" exitCode=0 Jul 10 17:32:52.785190 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.785025 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" Jul 10 17:32:52.785190 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.785030 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" event={"ID":"d8468276-c4a9-4170-a668-c463981e5b52","Type":"ContainerDied","Data":"84d59b5416c1645705ec5ec5cdf3c624b4a10282a6bb3cfbc40ab188644f1562"} Jul 10 17:32:52.785190 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.785071 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh" event={"ID":"d8468276-c4a9-4170-a668-c463981e5b52","Type":"ContainerDied","Data":"c93f3a162df2cc48c866664be7f664e2a778fc850ec7a39bf25eb426f11bd284"} Jul 10 17:32:52.785190 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.785095 2559 scope.go:117] "RemoveContainer" containerID="84d59b5416c1645705ec5ec5cdf3c624b4a10282a6bb3cfbc40ab188644f1562" Jul 10 17:32:52.794296 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.794274 2559 scope.go:117] "RemoveContainer" containerID="84d59b5416c1645705ec5ec5cdf3c624b4a10282a6bb3cfbc40ab188644f1562" Jul 10 17:32:52.794648 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:32:52.794626 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d59b5416c1645705ec5ec5cdf3c624b4a10282a6bb3cfbc40ab188644f1562\": container with ID starting with 84d59b5416c1645705ec5ec5cdf3c624b4a10282a6bb3cfbc40ab188644f1562 not found: ID does not exist" containerID="84d59b5416c1645705ec5ec5cdf3c624b4a10282a6bb3cfbc40ab188644f1562" Jul 10 17:32:52.794715 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.794659 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d59b5416c1645705ec5ec5cdf3c624b4a10282a6bb3cfbc40ab188644f1562"} err="failed to get container status \"84d59b5416c1645705ec5ec5cdf3c624b4a10282a6bb3cfbc40ab188644f1562\": rpc error: code = NotFound desc = could not find container \"84d59b5416c1645705ec5ec5cdf3c624b4a10282a6bb3cfbc40ab188644f1562\": container with ID starting with 84d59b5416c1645705ec5ec5cdf3c624b4a10282a6bb3cfbc40ab188644f1562 not found: ID does not exist" Jul 10 17:32:52.806971 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.806945 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvnm2\" (UniqueName: \"kubernetes.io/projected/d8468276-c4a9-4170-a668-c463981e5b52-kube-api-access-cvnm2\") pod \"d8468276-c4a9-4170-a668-c463981e5b52\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " Jul 10 17:32:52.807089 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.807061 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-model-cache\") pod \"d8468276-c4a9-4170-a668-c463981e5b52\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " Jul 10 17:32:52.807157 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.807090 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-home\") pod \"d8468276-c4a9-4170-a668-c463981e5b52\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " Jul 10 17:32:52.807157 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.807137 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-tmp-dir\") pod \"d8468276-c4a9-4170-a668-c463981e5b52\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " Jul 10 17:32:52.807259 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.807185 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8468276-c4a9-4170-a668-c463981e5b52-tls-certs\") pod \"d8468276-c4a9-4170-a668-c463981e5b52\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " Jul 10 17:32:52.807315 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.807264 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-dshm\") pod \"d8468276-c4a9-4170-a668-c463981e5b52\" (UID: \"d8468276-c4a9-4170-a668-c463981e5b52\") " Jul 10 17:32:52.807381 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.807354 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-home" (OuterVolumeSpecName: "home") pod "d8468276-c4a9-4170-a668-c463981e5b52" (UID: "d8468276-c4a9-4170-a668-c463981e5b52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:32:52.807381 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.807368 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8468276-c4a9-4170-a668-c463981e5b52" (UID: "d8468276-c4a9-4170-a668-c463981e5b52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:32:52.807476 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.807315 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8468276-c4a9-4170-a668-c463981e5b52" (UID: "d8468276-c4a9-4170-a668-c463981e5b52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:32:52.807602 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.807578 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-model-cache\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:32:52.807677 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.807608 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-home\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:32:52.807677 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.807629 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-tmp-dir\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:32:52.809479 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.809453 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8468276-c4a9-4170-a668-c463981e5b52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8468276-c4a9-4170-a668-c463981e5b52" (UID: "d8468276-c4a9-4170-a668-c463981e5b52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:32:52.809479 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.809467 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-dshm" (OuterVolumeSpecName: "dshm") pod "d8468276-c4a9-4170-a668-c463981e5b52" (UID: "d8468276-c4a9-4170-a668-c463981e5b52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:32:52.809479 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.809466 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8468276-c4a9-4170-a668-c463981e5b52-kube-api-access-cvnm2" (OuterVolumeSpecName: "kube-api-access-cvnm2") pod "d8468276-c4a9-4170-a668-c463981e5b52" (UID: "d8468276-c4a9-4170-a668-c463981e5b52"). InnerVolumeSpecName "kube-api-access-cvnm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:32:52.909011 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.908971 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8468276-c4a9-4170-a668-c463981e5b52-tls-certs\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:32:52.909011 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.909002 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8468276-c4a9-4170-a668-c463981e5b52-dshm\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:32:52.909011 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:52.909012 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvnm2\" (UniqueName: \"kubernetes.io/projected/d8468276-c4a9-4170-a668-c463981e5b52-kube-api-access-cvnm2\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:32:53.110964 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:53.110925 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh"] Jul 10 17:32:53.115036 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:53.115003 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-43091a/auth-test-model-43091a-kserve-ff58b45f9-s4qkh"] Jul 10 17:32:54.398855 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:32:54.398815 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8468276-c4a9-4170-a668-c463981e5b52" path="/var/lib/kubelet/pods/d8468276-c4a9-4170-a668-c463981e5b52/volumes" Jul 10 17:33:03.638652 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.638612 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf"] Jul 10 17:33:03.639103 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.639063 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8468276-c4a9-4170-a668-c463981e5b52" containerName="main" Jul 10 17:33:03.639103 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.639076 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8468276-c4a9-4170-a668-c463981e5b52" containerName="main" Jul 10 17:33:03.639179 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.639150 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8468276-c4a9-4170-a668-c463981e5b52" containerName="main" Jul 10 17:33:03.642437 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.642417 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.645151 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.645131 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2482f8\"/\"kube-root-ca.crt\"" Jul 10 17:33:03.646459 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.646341 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2482f8\"/\"sub-test-model-2482f8-kserve-self-signed-certs\"" Jul 10 17:33:03.646459 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.646356 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2482f8\"/\"openshift-service-ca.crt\"" Jul 10 17:33:03.646771 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.646751 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2482f8\"/\"default-dockercfg-8hgnp\"" Jul 10 17:33:03.649878 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.649649 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf"] Jul 10 17:33:03.664192 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.664163 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:33:03.700398 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.700366 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-dshm\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.700662 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.700416 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-home\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.700662 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.700434 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-model-cache\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.700662 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.700596 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-tmp-dir\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.700662 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.700627 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3916b05c-1f02-468d-9044-0989972e4df9-tls-certs\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.700662 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.700650 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcnwt\" (UniqueName: \"kubernetes.io/projected/3916b05c-1f02-468d-9044-0989972e4df9-kube-api-access-qcnwt\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.802010 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.801972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-dshm\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.802208 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.802039 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-home\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.802208 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.802060 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-model-cache\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.802208 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.802105 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-tmp-dir\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.802208 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.802128 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3916b05c-1f02-468d-9044-0989972e4df9-tls-certs\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.802208 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.802149 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcnwt\" (UniqueName: \"kubernetes.io/projected/3916b05c-1f02-468d-9044-0989972e4df9-kube-api-access-qcnwt\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.802677 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.802649 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-tmp-dir\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.802882 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.802671 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-home\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.802985 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.802929 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-model-cache\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.804506 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.804462 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-dshm\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.804813 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.804782 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3916b05c-1f02-468d-9044-0989972e4df9-tls-certs\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.810710 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.810681 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcnwt\" (UniqueName: \"kubernetes.io/projected/3916b05c-1f02-468d-9044-0989972e4df9-kube-api-access-qcnwt\") pod \"sub-test-model-2482f8-kserve-64b9f66475-84kpf\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:03.952884 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:03.952788 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:04.100178 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:04.100143 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf"] Jul 10 17:33:04.101649 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:33:04.101623 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3916b05c_1f02_468d_9044_0989972e4df9.slice/crio-231720269cfe33b861a0bbf0bb5e3e6d4d3a889909d849573f51e0a3bba1cb21 WatchSource:0}: Error finding container 231720269cfe33b861a0bbf0bb5e3e6d4d3a889909d849573f51e0a3bba1cb21: Status 404 returned error can't find the container with id 231720269cfe33b861a0bbf0bb5e3e6d4d3a889909d849573f51e0a3bba1cb21 Jul 10 17:33:04.853826 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:04.853790 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" event={"ID":"3916b05c-1f02-468d-9044-0989972e4df9","Type":"ContainerStarted","Data":"ee6f92a9c89534f0a12c51b44c457615935dff5c2dc50dbdbb2103f5f4e80be1"} Jul 10 17:33:04.853826 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:04.853831 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" event={"ID":"3916b05c-1f02-468d-9044-0989972e4df9","Type":"ContainerStarted","Data":"231720269cfe33b861a0bbf0bb5e3e6d4d3a889909d849573f51e0a3bba1cb21"} Jul 10 17:33:04.870040 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:04.869986 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" podStartSLOduration=1.869968848 podStartE2EDuration="1.869968848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:33:04.868750702 +0000 UTC m=+2324.994459124" watchObservedRunningTime="2026-07-10 17:33:04.869968848 +0000 UTC m=+2324.995677268" Jul 10 17:33:10.232886 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:10.232845 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:33:13.953206 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:13.953167 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:13.953650 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:13.953252 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:13.965644 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:13.965621 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:14.918153 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:14.918123 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:33:25.535042 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:25.535001 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:33:36.096353 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:36.096314 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:33:36.197744 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:36.197708 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:33:46.333257 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:46.333218 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:33:51.368426 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:51.368391 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:33:52.146056 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:52.146019 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf"] Jul 10 17:33:52.161413 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:33:52.161381 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:34:00.975122 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:00.975083 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:34:05.134079 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:05.134045 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:34:07.170063 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.169998 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" podUID="3916b05c-1f02-468d-9044-0989972e4df9" containerName="main" containerID="cri-o://ee6f92a9c89534f0a12c51b44c457615935dff5c2dc50dbdbb2103f5f4e80be1" gracePeriod=45 Jul 10 17:34:07.431308 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.431243 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:34:07.504764 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.504732 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-model-cache\") pod \"3916b05c-1f02-468d-9044-0989972e4df9\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " Jul 10 17:34:07.504948 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.504779 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-dshm\") pod \"3916b05c-1f02-468d-9044-0989972e4df9\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " Jul 10 17:34:07.504948 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.504798 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-tmp-dir\") pod \"3916b05c-1f02-468d-9044-0989972e4df9\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " Jul 10 17:34:07.504948 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.504830 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3916b05c-1f02-468d-9044-0989972e4df9-tls-certs\") pod \"3916b05c-1f02-468d-9044-0989972e4df9\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " Jul 10 17:34:07.504948 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.504880 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcnwt\" (UniqueName: \"kubernetes.io/projected/3916b05c-1f02-468d-9044-0989972e4df9-kube-api-access-qcnwt\") pod \"3916b05c-1f02-468d-9044-0989972e4df9\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " Jul 10 17:34:07.504948 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.504896 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-home\") pod \"3916b05c-1f02-468d-9044-0989972e4df9\" (UID: \"3916b05c-1f02-468d-9044-0989972e4df9\") " Jul 10 17:34:07.505231 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.505100 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3916b05c-1f02-468d-9044-0989972e4df9" (UID: "3916b05c-1f02-468d-9044-0989972e4df9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:34:07.505231 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.505116 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-model-cache" (OuterVolumeSpecName: "model-cache") pod "3916b05c-1f02-468d-9044-0989972e4df9" (UID: "3916b05c-1f02-468d-9044-0989972e4df9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:34:07.505231 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.505208 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-model-cache\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:34:07.505231 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.505226 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-tmp-dir\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:34:07.505416 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.505322 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-home" (OuterVolumeSpecName: "home") pod "3916b05c-1f02-468d-9044-0989972e4df9" (UID: "3916b05c-1f02-468d-9044-0989972e4df9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:34:07.507313 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.507285 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3916b05c-1f02-468d-9044-0989972e4df9-kube-api-access-qcnwt" (OuterVolumeSpecName: "kube-api-access-qcnwt") pod "3916b05c-1f02-468d-9044-0989972e4df9" (UID: "3916b05c-1f02-468d-9044-0989972e4df9"). InnerVolumeSpecName "kube-api-access-qcnwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:34:07.507567 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.507541 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-dshm" (OuterVolumeSpecName: "dshm") pod "3916b05c-1f02-468d-9044-0989972e4df9" (UID: "3916b05c-1f02-468d-9044-0989972e4df9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:34:07.507680 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.507544 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3916b05c-1f02-468d-9044-0989972e4df9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3916b05c-1f02-468d-9044-0989972e4df9" (UID: "3916b05c-1f02-468d-9044-0989972e4df9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:34:07.605901 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.605860 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-dshm\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:34:07.605901 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.605893 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3916b05c-1f02-468d-9044-0989972e4df9-tls-certs\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:34:07.605901 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.605905 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcnwt\" (UniqueName: \"kubernetes.io/projected/3916b05c-1f02-468d-9044-0989972e4df9-kube-api-access-qcnwt\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:34:07.606126 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:07.605914 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3916b05c-1f02-468d-9044-0989972e4df9-home\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:34:08.194077 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:08.194046 2559 generic.go:356] "Generic (PLEG): container finished" podID="3916b05c-1f02-468d-9044-0989972e4df9" containerID="ee6f92a9c89534f0a12c51b44c457615935dff5c2dc50dbdbb2103f5f4e80be1" exitCode=0 Jul 10 17:34:08.194520 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:08.194090 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" Jul 10 17:34:08.194520 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:08.194135 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" event={"ID":"3916b05c-1f02-468d-9044-0989972e4df9","Type":"ContainerDied","Data":"ee6f92a9c89534f0a12c51b44c457615935dff5c2dc50dbdbb2103f5f4e80be1"} Jul 10 17:34:08.194520 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:08.194181 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf" event={"ID":"3916b05c-1f02-468d-9044-0989972e4df9","Type":"ContainerDied","Data":"231720269cfe33b861a0bbf0bb5e3e6d4d3a889909d849573f51e0a3bba1cb21"} Jul 10 17:34:08.194520 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:08.194204 2559 scope.go:117] "RemoveContainer" containerID="ee6f92a9c89534f0a12c51b44c457615935dff5c2dc50dbdbb2103f5f4e80be1" Jul 10 17:34:08.204685 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:08.204665 2559 scope.go:117] "RemoveContainer" containerID="ee6f92a9c89534f0a12c51b44c457615935dff5c2dc50dbdbb2103f5f4e80be1" Jul 10 17:34:08.204962 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:34:08.204940 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6f92a9c89534f0a12c51b44c457615935dff5c2dc50dbdbb2103f5f4e80be1\": container with ID starting with ee6f92a9c89534f0a12c51b44c457615935dff5c2dc50dbdbb2103f5f4e80be1 not found: ID does not exist" containerID="ee6f92a9c89534f0a12c51b44c457615935dff5c2dc50dbdbb2103f5f4e80be1" Jul 10 17:34:08.205028 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:08.204970 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6f92a9c89534f0a12c51b44c457615935dff5c2dc50dbdbb2103f5f4e80be1"} err="failed to get container status \"ee6f92a9c89534f0a12c51b44c457615935dff5c2dc50dbdbb2103f5f4e80be1\": rpc error: code = NotFound desc = could not find container \"ee6f92a9c89534f0a12c51b44c457615935dff5c2dc50dbdbb2103f5f4e80be1\": container with ID starting with ee6f92a9c89534f0a12c51b44c457615935dff5c2dc50dbdbb2103f5f4e80be1 not found: ID does not exist" Jul 10 17:34:08.217120 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:08.217084 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf"] Jul 10 17:34:08.220344 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:08.220316 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2482f8/sub-test-model-2482f8-kserve-64b9f66475-84kpf"] Jul 10 17:34:08.401448 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:08.401409 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3916b05c-1f02-468d-9044-0989972e4df9" path="/var/lib/kubelet/pods/3916b05c-1f02-468d-9044-0989972e4df9/volumes" Jul 10 17:34:31.834726 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.834695 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb"] Jul 10 17:34:31.835173 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.835070 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3916b05c-1f02-468d-9044-0989972e4df9" containerName="main" Jul 10 17:34:31.835173 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.835080 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3916b05c-1f02-468d-9044-0989972e4df9" containerName="main" Jul 10 17:34:31.835173 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.835151 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3916b05c-1f02-468d-9044-0989972e4df9" containerName="main" Jul 10 17:34:31.837989 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.837967 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:31.841471 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.841437 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2482f8\"/\"default-dockercfg-8hgnp\"" Jul 10 17:34:31.841471 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.841455 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2482f8\"/\"openshift-service-ca.crt\"" Jul 10 17:34:31.841471 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.841468 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2482f8\"/\"rate-test-model-2482f8-kserve-self-signed-certs\"" Jul 10 17:34:31.841713 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.841549 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2482f8\"/\"kube-root-ca.crt\"" Jul 10 17:34:31.845411 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.845386 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb"] Jul 10 17:34:31.926396 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.926362 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-dshm\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:31.926613 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.926422 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-home\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:31.926613 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.926557 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-tmp-dir\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:31.926740 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.926621 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b1a912-8634-483a-808d-a83a3434908a-tls-certs\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:31.926740 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.926709 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjq4r\" (UniqueName: \"kubernetes.io/projected/98b1a912-8634-483a-808d-a83a3434908a-kube-api-access-cjq4r\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:31.926852 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.926779 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-model-cache\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:31.935167 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:31.935135 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:34:32.027303 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.027262 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-tmp-dir\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:32.027303 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.027308 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b1a912-8634-483a-808d-a83a3434908a-tls-certs\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:32.027614 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.027332 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjq4r\" (UniqueName: \"kubernetes.io/projected/98b1a912-8634-483a-808d-a83a3434908a-kube-api-access-cjq4r\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:32.027614 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.027355 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-model-cache\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:32.027614 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.027377 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-dshm\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:32.027614 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.027516 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-home\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:32.027803 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.027743 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-tmp-dir\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:32.027803 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.027777 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-model-cache\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:32.027880 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.027854 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-home\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:32.029844 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.029824 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-dshm\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:32.030082 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.030066 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b1a912-8634-483a-808d-a83a3434908a-tls-certs\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:32.035311 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.035286 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjq4r\" (UniqueName: \"kubernetes.io/projected/98b1a912-8634-483a-808d-a83a3434908a-kube-api-access-cjq4r\") pod \"rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:32.148546 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.148473 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:32.288092 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.288059 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb"] Jul 10 17:34:32.289041 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:34:32.289014 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98b1a912_8634_483a_808d_a83a3434908a.slice/crio-b2c773858f9bf1ffcd871ad227c9c02a9e8fc10650c69acd59faccbfbac1b025 WatchSource:0}: Error finding container b2c773858f9bf1ffcd871ad227c9c02a9e8fc10650c69acd59faccbfbac1b025: Status 404 returned error can't find the container with id b2c773858f9bf1ffcd871ad227c9c02a9e8fc10650c69acd59faccbfbac1b025 Jul 10 17:34:32.327966 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:32.327929 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" event={"ID":"98b1a912-8634-483a-808d-a83a3434908a","Type":"ContainerStarted","Data":"b2c773858f9bf1ffcd871ad227c9c02a9e8fc10650c69acd59faccbfbac1b025"} Jul 10 17:34:33.335377 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:33.335343 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" event={"ID":"98b1a912-8634-483a-808d-a83a3434908a","Type":"ContainerStarted","Data":"c12231de8c725a0997a658956846012392119efc88a262a84eb0393a35fd90b4"} Jul 10 17:34:33.354511 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:33.354441 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" podStartSLOduration=2.354426223 podStartE2EDuration="2.354426223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:34:33.35234491 +0000 UTC m=+2413.478053334" watchObservedRunningTime="2026-07-10 17:34:33.354426223 +0000 UTC m=+2413.480134633" Jul 10 17:34:36.945829 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:36.945794 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:34:37.028477 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:37.028438 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:34:37.029841 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:37.029812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:34:42.149114 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:42.149024 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:42.149114 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:42.149092 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:42.161670 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:42.161643 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:42.400828 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:42.400734 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:34:44.162026 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:44.161989 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:34:44.259435 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:44.259402 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:34:50.399596 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:50.399560 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:34:54.040042 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:54.040010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:34:54.141211 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:34:54.141175 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:01.297138 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:01.297100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:19.532967 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:19.532931 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:35:35.234648 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:35.234610 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:35:41.610250 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:41.610213 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:42.527229 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:42.527187 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb"] Jul 10 17:35:42.537930 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:42.537896 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:43.514826 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:43.514790 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7463-140489726480176"] Jul 10 17:35:43.518324 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:43.518302 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726480176" Jul 10 17:35:43.521180 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:43.521149 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-fg4vv\"" Jul 10 17:35:43.523549 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:43.523525 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7463-140489726480176"] Jul 10 17:35:43.605391 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:43.605354 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cr76\" (UniqueName: \"kubernetes.io/projected/ea2dcf4e-f111-432c-9a8e-48223497adc5-kube-api-access-6cr76\") pod \"test-curl-7463-140489726480176\" (UID: \"ea2dcf4e-f111-432c-9a8e-48223497adc5\") " pod="openshift-ingress/test-curl-7463-140489726480176" Jul 10 17:35:43.706870 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:43.706813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cr76\" (UniqueName: \"kubernetes.io/projected/ea2dcf4e-f111-432c-9a8e-48223497adc5-kube-api-access-6cr76\") pod \"test-curl-7463-140489726480176\" (UID: \"ea2dcf4e-f111-432c-9a8e-48223497adc5\") " pod="openshift-ingress/test-curl-7463-140489726480176" Jul 10 17:35:43.715999 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:43.715974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cr76\" (UniqueName: \"kubernetes.io/projected/ea2dcf4e-f111-432c-9a8e-48223497adc5-kube-api-access-6cr76\") pod \"test-curl-7463-140489726480176\" (UID: \"ea2dcf4e-f111-432c-9a8e-48223497adc5\") " pod="openshift-ingress/test-curl-7463-140489726480176" Jul 10 17:35:43.830478 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:43.830380 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726480176" Jul 10 17:35:43.974510 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:43.974466 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7463-140489726480176"] Jul 10 17:35:43.976046 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:35:43.976019 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea2dcf4e_f111_432c_9a8e_48223497adc5.slice/crio-2693e38126aa0e28568b6709dbac43a419985a1f4ec33bdd0aa048c306fb9c27 WatchSource:0}: Error finding container 2693e38126aa0e28568b6709dbac43a419985a1f4ec33bdd0aa048c306fb9c27: Status 404 returned error can't find the container with id 2693e38126aa0e28568b6709dbac43a419985a1f4ec33bdd0aa048c306fb9c27 Jul 10 17:35:43.977471 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:43.977447 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:35:44.738766 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:44.738723 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7463-140489726480176" event={"ID":"ea2dcf4e-f111-432c-9a8e-48223497adc5","Type":"ContainerStarted","Data":"2693e38126aa0e28568b6709dbac43a419985a1f4ec33bdd0aa048c306fb9c27"} Jul 10 17:35:45.745326 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:45.745288 2559 generic.go:356] "Generic (PLEG): container finished" podID="ea2dcf4e-f111-432c-9a8e-48223497adc5" containerID="b27ac2320fafb20d89d8a1d03690b59f1a1c458e8c6190cd1bc7d04e815ee8ae" exitCode=0 Jul 10 17:35:45.745749 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:45.745340 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7463-140489726480176" event={"ID":"ea2dcf4e-f111-432c-9a8e-48223497adc5","Type":"ContainerDied","Data":"b27ac2320fafb20d89d8a1d03690b59f1a1c458e8c6190cd1bc7d04e815ee8ae"} Jul 10 17:35:46.892081 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:46.892053 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726480176" Jul 10 17:35:46.910423 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:46.910387 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7463-140489726480176_ea2dcf4e-f111-432c-9a8e-48223497adc5/test-curl-7463-140489726480176/0.log" Jul 10 17:35:46.935238 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:46.935205 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7463-140489726480176"] Jul 10 17:35:46.937528 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:46.937506 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7463-140489726480176"] Jul 10 17:35:47.049098 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:47.049010 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cr76\" (UniqueName: \"kubernetes.io/projected/ea2dcf4e-f111-432c-9a8e-48223497adc5-kube-api-access-6cr76\") pod \"ea2dcf4e-f111-432c-9a8e-48223497adc5\" (UID: \"ea2dcf4e-f111-432c-9a8e-48223497adc5\") " Jul 10 17:35:47.051264 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:47.051235 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2dcf4e-f111-432c-9a8e-48223497adc5-kube-api-access-6cr76" (OuterVolumeSpecName: "kube-api-access-6cr76") pod "ea2dcf4e-f111-432c-9a8e-48223497adc5" (UID: "ea2dcf4e-f111-432c-9a8e-48223497adc5"). InnerVolumeSpecName "kube-api-access-6cr76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:35:47.150447 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:47.150405 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cr76\" (UniqueName: \"kubernetes.io/projected/ea2dcf4e-f111-432c-9a8e-48223497adc5-kube-api-access-6cr76\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:35:47.758106 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:47.758074 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2693e38126aa0e28568b6709dbac43a419985a1f4ec33bdd0aa048c306fb9c27" Jul 10 17:35:47.758106 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:47.758097 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726480176" Jul 10 17:35:47.769168 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:35:47.769133 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7463-140489726480176\" is forbidden: User \"system:node:ip-10-0-141-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-228.ec2.internal' and this object" podUID="ea2dcf4e-f111-432c-9a8e-48223497adc5" pod="openshift-ingress/test-curl-7463-140489726480176" Jul 10 17:35:48.398702 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:48.398670 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea2dcf4e-f111-432c-9a8e-48223497adc5" path="/var/lib/kubelet/pods/ea2dcf4e-f111-432c-9a8e-48223497adc5/volumes" Jul 10 17:35:50.074828 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:50.074782 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7463-140489726516224"] Jul 10 17:35:50.075394 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:50.075372 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2dcf4e-f111-432c-9a8e-48223497adc5" containerName="test-curl-7463-140489726480176" Jul 10 17:35:50.075485 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:50.075397 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2dcf4e-f111-432c-9a8e-48223497adc5" containerName="test-curl-7463-140489726480176" Jul 10 17:35:50.075587 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:50.075517 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2dcf4e-f111-432c-9a8e-48223497adc5" containerName="test-curl-7463-140489726480176" Jul 10 17:35:50.078726 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:50.078704 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726516224" Jul 10 17:35:50.087670 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:50.087466 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-fg4vv\"" Jul 10 17:35:50.090058 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:50.090029 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7463-140489726516224"] Jul 10 17:35:50.181068 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:50.181032 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6svfr\" (UniqueName: \"kubernetes.io/projected/d874ad3a-9166-4ba4-a4e8-04784be1e705-kube-api-access-6svfr\") pod \"test-curl-7463-140489726516224\" (UID: \"d874ad3a-9166-4ba4-a4e8-04784be1e705\") " pod="openshift-ingress/test-curl-7463-140489726516224" Jul 10 17:35:50.281657 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:50.281615 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6svfr\" (UniqueName: \"kubernetes.io/projected/d874ad3a-9166-4ba4-a4e8-04784be1e705-kube-api-access-6svfr\") pod \"test-curl-7463-140489726516224\" (UID: \"d874ad3a-9166-4ba4-a4e8-04784be1e705\") " pod="openshift-ingress/test-curl-7463-140489726516224" Jul 10 17:35:50.290360 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:50.290328 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6svfr\" (UniqueName: \"kubernetes.io/projected/d874ad3a-9166-4ba4-a4e8-04784be1e705-kube-api-access-6svfr\") pod \"test-curl-7463-140489726516224\" (UID: \"d874ad3a-9166-4ba4-a4e8-04784be1e705\") " pod="openshift-ingress/test-curl-7463-140489726516224" Jul 10 17:35:50.395370 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:50.395336 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726516224" Jul 10 17:35:50.528330 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:50.528307 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7463-140489726516224"] Jul 10 17:35:50.530182 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:35:50.530153 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd874ad3a_9166_4ba4_a4e8_04784be1e705.slice/crio-dc266bed1492ede5300a5977f84307f37ed9b4d14ea9d1de7afdbe13ebc81f44 WatchSource:0}: Error finding container dc266bed1492ede5300a5977f84307f37ed9b4d14ea9d1de7afdbe13ebc81f44: Status 404 returned error can't find the container with id dc266bed1492ede5300a5977f84307f37ed9b4d14ea9d1de7afdbe13ebc81f44 Jul 10 17:35:50.774441 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:50.774354 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7463-140489726516224" event={"ID":"d874ad3a-9166-4ba4-a4e8-04784be1e705","Type":"ContainerStarted","Data":"dc266bed1492ede5300a5977f84307f37ed9b4d14ea9d1de7afdbe13ebc81f44"} Jul 10 17:35:51.781620 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:51.781584 2559 generic.go:356] "Generic (PLEG): container finished" podID="d874ad3a-9166-4ba4-a4e8-04784be1e705" containerID="8e6c7002fba3d5505cbec92b4880abc08d79f04e5a0052deb426cd0124b17b54" exitCode=0 Jul 10 17:35:51.781999 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:51.781677 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7463-140489726516224" event={"ID":"d874ad3a-9166-4ba4-a4e8-04784be1e705","Type":"ContainerDied","Data":"8e6c7002fba3d5505cbec92b4880abc08d79f04e5a0052deb426cd0124b17b54"} Jul 10 17:35:52.934147 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:52.934119 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726516224" Jul 10 17:35:52.952975 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:52.952949 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7463-140489726516224_d874ad3a-9166-4ba4-a4e8-04784be1e705/test-curl-7463-140489726516224/0.log" Jul 10 17:35:52.974378 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:52.974336 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7463-140489726516224"] Jul 10 17:35:52.978195 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:52.978170 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7463-140489726516224"] Jul 10 17:35:53.003587 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:53.003563 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6svfr\" (UniqueName: \"kubernetes.io/projected/d874ad3a-9166-4ba4-a4e8-04784be1e705-kube-api-access-6svfr\") pod \"d874ad3a-9166-4ba4-a4e8-04784be1e705\" (UID: \"d874ad3a-9166-4ba4-a4e8-04784be1e705\") " Jul 10 17:35:53.005775 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:53.005744 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d874ad3a-9166-4ba4-a4e8-04784be1e705-kube-api-access-6svfr" (OuterVolumeSpecName: "kube-api-access-6svfr") pod "d874ad3a-9166-4ba4-a4e8-04784be1e705" (UID: "d874ad3a-9166-4ba4-a4e8-04784be1e705"). InnerVolumeSpecName "kube-api-access-6svfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:35:53.104278 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:53.104238 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6svfr\" (UniqueName: \"kubernetes.io/projected/d874ad3a-9166-4ba4-a4e8-04784be1e705-kube-api-access-6svfr\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:35:53.794591 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:53.794552 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726516224" Jul 10 17:35:53.794591 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:53.794570 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc266bed1492ede5300a5977f84307f37ed9b4d14ea9d1de7afdbe13ebc81f44" Jul 10 17:35:53.805023 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:35:53.804994 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7463-140489726516224\" is forbidden: User \"system:node:ip-10-0-141-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-228.ec2.internal' and this object" podUID="d874ad3a-9166-4ba4-a4e8-04784be1e705" pod="openshift-ingress/test-curl-7463-140489726516224" Jul 10 17:35:54.036204 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:54.036165 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:35:54.399776 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:54.399734 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d874ad3a-9166-4ba4-a4e8-04784be1e705" path="/var/lib/kubelet/pods/d874ad3a-9166-4ba4-a4e8-04784be1e705/volumes" Jul 10 17:35:56.053377 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:56.053339 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7463-140489726517088"] Jul 10 17:35:56.053799 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:56.053765 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d874ad3a-9166-4ba4-a4e8-04784be1e705" containerName="test-curl-7463-140489726516224" Jul 10 17:35:56.053799 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:56.053777 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d874ad3a-9166-4ba4-a4e8-04784be1e705" containerName="test-curl-7463-140489726516224" Jul 10 17:35:56.053870 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:56.053850 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d874ad3a-9166-4ba4-a4e8-04784be1e705" containerName="test-curl-7463-140489726516224" Jul 10 17:35:56.056902 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:56.056885 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:35:56.059336 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:56.059318 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-fg4vv\"" Jul 10 17:35:56.063403 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:56.063378 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7463-140489726517088"] Jul 10 17:35:56.128816 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:56.128777 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk659\" (UniqueName: \"kubernetes.io/projected/2d30e365-51d6-4ce8-92b4-80945af7ba40-kube-api-access-qk659\") pod \"test-curl-7463-140489726517088\" (UID: \"2d30e365-51d6-4ce8-92b4-80945af7ba40\") " pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:35:56.229659 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:56.229621 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk659\" (UniqueName: \"kubernetes.io/projected/2d30e365-51d6-4ce8-92b4-80945af7ba40-kube-api-access-qk659\") pod \"test-curl-7463-140489726517088\" (UID: \"2d30e365-51d6-4ce8-92b4-80945af7ba40\") " pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:35:56.238923 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:56.238887 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk659\" (UniqueName: \"kubernetes.io/projected/2d30e365-51d6-4ce8-92b4-80945af7ba40-kube-api-access-qk659\") pod \"test-curl-7463-140489726517088\" (UID: \"2d30e365-51d6-4ce8-92b4-80945af7ba40\") " pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:35:56.367165 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:56.367127 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:35:56.512366 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:56.512338 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7463-140489726517088"] Jul 10 17:35:56.513652 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:35:56.513616 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d30e365_51d6_4ce8_92b4_80945af7ba40.slice/crio-cc978cc1ad6ad09f07d2e36adbd06a152dcd04ffeef61eb31739a45e4d5fc03a WatchSource:0}: Error finding container cc978cc1ad6ad09f07d2e36adbd06a152dcd04ffeef61eb31739a45e4d5fc03a: Status 404 returned error can't find the container with id cc978cc1ad6ad09f07d2e36adbd06a152dcd04ffeef61eb31739a45e4d5fc03a Jul 10 17:35:56.818814 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:56.818730 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7463-140489726517088" event={"ID":"2d30e365-51d6-4ce8-92b4-80945af7ba40","Type":"ContainerStarted","Data":"cc978cc1ad6ad09f07d2e36adbd06a152dcd04ffeef61eb31739a45e4d5fc03a"} Jul 10 17:35:57.549017 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.548894 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" podUID="98b1a912-8634-483a-808d-a83a3434908a" containerName="main" containerID="cri-o://c12231de8c725a0997a658956846012392119efc88a262a84eb0393a35fd90b4" gracePeriod=45 Jul 10 17:35:57.812885 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.812821 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:35:57.825763 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.825731 2559 generic.go:356] "Generic (PLEG): container finished" podID="2d30e365-51d6-4ce8-92b4-80945af7ba40" containerID="b027ab5e1938cbfd489c7bcd89703f47f093caba2109b94c4e1d6568dd0fc059" exitCode=0 Jul 10 17:35:57.825923 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.825819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7463-140489726517088" event={"ID":"2d30e365-51d6-4ce8-92b4-80945af7ba40","Type":"ContainerDied","Data":"b027ab5e1938cbfd489c7bcd89703f47f093caba2109b94c4e1d6568dd0fc059"} Jul 10 17:35:57.827226 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.827203 2559 generic.go:356] "Generic (PLEG): container finished" podID="98b1a912-8634-483a-808d-a83a3434908a" containerID="c12231de8c725a0997a658956846012392119efc88a262a84eb0393a35fd90b4" exitCode=0 Jul 10 17:35:57.827345 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.827233 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" Jul 10 17:35:57.827345 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.827248 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" event={"ID":"98b1a912-8634-483a-808d-a83a3434908a","Type":"ContainerDied","Data":"c12231de8c725a0997a658956846012392119efc88a262a84eb0393a35fd90b4"} Jul 10 17:35:57.827345 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.827279 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb" event={"ID":"98b1a912-8634-483a-808d-a83a3434908a","Type":"ContainerDied","Data":"b2c773858f9bf1ffcd871ad227c9c02a9e8fc10650c69acd59faccbfbac1b025"} Jul 10 17:35:57.827345 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.827301 2559 scope.go:117] "RemoveContainer" containerID="c12231de8c725a0997a658956846012392119efc88a262a84eb0393a35fd90b4" Jul 10 17:35:57.839014 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.838994 2559 scope.go:117] "RemoveContainer" containerID="c12231de8c725a0997a658956846012392119efc88a262a84eb0393a35fd90b4" Jul 10 17:35:57.839300 ip-10-0-141-228 kubenswrapper[2559]: E0710 17:35:57.839277 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12231de8c725a0997a658956846012392119efc88a262a84eb0393a35fd90b4\": container with ID starting with c12231de8c725a0997a658956846012392119efc88a262a84eb0393a35fd90b4 not found: ID does not exist" containerID="c12231de8c725a0997a658956846012392119efc88a262a84eb0393a35fd90b4" Jul 10 17:35:57.839370 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.839307 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12231de8c725a0997a658956846012392119efc88a262a84eb0393a35fd90b4"} err="failed to get container status \"c12231de8c725a0997a658956846012392119efc88a262a84eb0393a35fd90b4\": rpc error: code = NotFound desc = could not find container \"c12231de8c725a0997a658956846012392119efc88a262a84eb0393a35fd90b4\": container with ID starting with c12231de8c725a0997a658956846012392119efc88a262a84eb0393a35fd90b4 not found: ID does not exist" Jul 10 17:35:57.843714 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.843694 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-dshm\") pod \"98b1a912-8634-483a-808d-a83a3434908a\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " Jul 10 17:35:57.843791 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.843722 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-home\") pod \"98b1a912-8634-483a-808d-a83a3434908a\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " Jul 10 17:35:57.843791 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.843738 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b1a912-8634-483a-808d-a83a3434908a-tls-certs\") pod \"98b1a912-8634-483a-808d-a83a3434908a\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " Jul 10 17:35:57.843791 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.843786 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjq4r\" (UniqueName: \"kubernetes.io/projected/98b1a912-8634-483a-808d-a83a3434908a-kube-api-access-cjq4r\") pod \"98b1a912-8634-483a-808d-a83a3434908a\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " Jul 10 17:35:57.843948 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.843869 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-model-cache\") pod \"98b1a912-8634-483a-808d-a83a3434908a\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " Jul 10 17:35:57.843948 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.843905 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-tmp-dir\") pod \"98b1a912-8634-483a-808d-a83a3434908a\" (UID: \"98b1a912-8634-483a-808d-a83a3434908a\") " Jul 10 17:35:57.844048 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.844009 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-home" (OuterVolumeSpecName: "home") pod "98b1a912-8634-483a-808d-a83a3434908a" (UID: "98b1a912-8634-483a-808d-a83a3434908a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:35:57.844152 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.844123 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-model-cache" (OuterVolumeSpecName: "model-cache") pod "98b1a912-8634-483a-808d-a83a3434908a" (UID: "98b1a912-8634-483a-808d-a83a3434908a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:35:57.844236 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.844215 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98b1a912-8634-483a-808d-a83a3434908a" (UID: "98b1a912-8634-483a-808d-a83a3434908a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:35:57.844331 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.844310 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-model-cache\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:35:57.844331 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.844332 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-tmp-dir\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:35:57.844535 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.844341 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-home\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:35:57.846093 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.846062 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b1a912-8634-483a-808d-a83a3434908a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98b1a912-8634-483a-808d-a83a3434908a" (UID: "98b1a912-8634-483a-808d-a83a3434908a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:35:57.846714 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.846677 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-dshm" (OuterVolumeSpecName: "dshm") pod "98b1a912-8634-483a-808d-a83a3434908a" (UID: "98b1a912-8634-483a-808d-a83a3434908a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:35:57.846993 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.846964 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b1a912-8634-483a-808d-a83a3434908a-kube-api-access-cjq4r" (OuterVolumeSpecName: "kube-api-access-cjq4r") pod "98b1a912-8634-483a-808d-a83a3434908a" (UID: "98b1a912-8634-483a-808d-a83a3434908a"). InnerVolumeSpecName "kube-api-access-cjq4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:35:57.945613 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.945570 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b1a912-8634-483a-808d-a83a3434908a-dshm\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:35:57.945613 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.945604 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b1a912-8634-483a-808d-a83a3434908a-tls-certs\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:35:57.945613 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:57.945619 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjq4r\" (UniqueName: \"kubernetes.io/projected/98b1a912-8634-483a-808d-a83a3434908a-kube-api-access-cjq4r\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:35:58.149373 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:58.149339 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb"] Jul 10 17:35:58.154919 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:58.154895 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2482f8/rate-test-model-2482f8-kserve-54fd6ff8c5-wlhbb"] Jul 10 17:35:58.398669 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:58.398584 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98b1a912-8634-483a-808d-a83a3434908a" path="/var/lib/kubelet/pods/98b1a912-8634-483a-808d-a83a3434908a/volumes" Jul 10 17:35:58.979962 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:58.979935 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:35:58.999280 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:58.999253 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7463-140489726517088_2d30e365-51d6-4ce8-92b4-80945af7ba40/test-curl-7463-140489726517088/0.log" Jul 10 17:35:59.024187 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.024129 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7463-140489726517088"] Jul 10 17:35:59.030483 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.030450 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7463-140489726517088"] Jul 10 17:35:59.056170 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.056130 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk659\" (UniqueName: \"kubernetes.io/projected/2d30e365-51d6-4ce8-92b4-80945af7ba40-kube-api-access-qk659\") pod \"2d30e365-51d6-4ce8-92b4-80945af7ba40\" (UID: \"2d30e365-51d6-4ce8-92b4-80945af7ba40\") " Jul 10 17:35:59.058396 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.058359 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d30e365-51d6-4ce8-92b4-80945af7ba40-kube-api-access-qk659" (OuterVolumeSpecName: "kube-api-access-qk659") pod "2d30e365-51d6-4ce8-92b4-80945af7ba40" (UID: "2d30e365-51d6-4ce8-92b4-80945af7ba40"). InnerVolumeSpecName "kube-api-access-qk659". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:35:59.146062 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.146022 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7463-140489726517088"] Jul 10 17:35:59.146449 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.146436 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b1a912-8634-483a-808d-a83a3434908a" containerName="main" Jul 10 17:35:59.146531 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.146450 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b1a912-8634-483a-808d-a83a3434908a" containerName="main" Jul 10 17:35:59.146531 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.146469 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d30e365-51d6-4ce8-92b4-80945af7ba40" containerName="test-curl-7463-140489726517088" Jul 10 17:35:59.146531 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.146475 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d30e365-51d6-4ce8-92b4-80945af7ba40" containerName="test-curl-7463-140489726517088" Jul 10 17:35:59.146650 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.146566 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d30e365-51d6-4ce8-92b4-80945af7ba40" containerName="test-curl-7463-140489726517088" Jul 10 17:35:59.146650 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.146582 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="98b1a912-8634-483a-808d-a83a3434908a" containerName="main" Jul 10 17:35:59.149726 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.149710 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:35:59.155282 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.155254 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7463-140489726517088"] Jul 10 17:35:59.157131 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.157100 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4dmt\" (UniqueName: \"kubernetes.io/projected/b617a4bd-63a8-42bd-a67f-7e72564ce56f-kube-api-access-j4dmt\") pod \"test-curl-7463-140489726517088\" (UID: \"b617a4bd-63a8-42bd-a67f-7e72564ce56f\") " pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:35:59.157348 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.157329 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk659\" (UniqueName: \"kubernetes.io/projected/2d30e365-51d6-4ce8-92b4-80945af7ba40-kube-api-access-qk659\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:35:59.258531 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.258470 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4dmt\" (UniqueName: \"kubernetes.io/projected/b617a4bd-63a8-42bd-a67f-7e72564ce56f-kube-api-access-j4dmt\") pod \"test-curl-7463-140489726517088\" (UID: \"b617a4bd-63a8-42bd-a67f-7e72564ce56f\") " pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:35:59.266883 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.266847 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4dmt\" (UniqueName: \"kubernetes.io/projected/b617a4bd-63a8-42bd-a67f-7e72564ce56f-kube-api-access-j4dmt\") pod \"test-curl-7463-140489726517088\" (UID: \"b617a4bd-63a8-42bd-a67f-7e72564ce56f\") " pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:35:59.460749 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.460649 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:35:59.592623 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.592593 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7463-140489726517088"] Jul 10 17:35:59.594201 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:35:59.594174 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb617a4bd_63a8_42bd_a67f_7e72564ce56f.slice/crio-9cfca9657d2fa7cc2d07c3ae3a692857dc0b6ac1db587d75c16cc271087289e0 WatchSource:0}: Error finding container 9cfca9657d2fa7cc2d07c3ae3a692857dc0b6ac1db587d75c16cc271087289e0: Status 404 returned error can't find the container with id 9cfca9657d2fa7cc2d07c3ae3a692857dc0b6ac1db587d75c16cc271087289e0 Jul 10 17:35:59.840966 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.840935 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:35:59.841150 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.840946 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc978cc1ad6ad09f07d2e36adbd06a152dcd04ffeef61eb31739a45e4d5fc03a" Jul 10 17:35:59.841956 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.841926 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7463-140489726517088" event={"ID":"b617a4bd-63a8-42bd-a67f-7e72564ce56f","Type":"ContainerStarted","Data":"9cfca9657d2fa7cc2d07c3ae3a692857dc0b6ac1db587d75c16cc271087289e0"} Jul 10 17:35:59.852830 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:35:59.852800 2559 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7463-140489726517088" oldPodUID="2d30e365-51d6-4ce8-92b4-80945af7ba40" podUID="b617a4bd-63a8-42bd-a67f-7e72564ce56f" Jul 10 17:36:00.399458 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:00.399424 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d30e365-51d6-4ce8-92b4-80945af7ba40" path="/var/lib/kubelet/pods/2d30e365-51d6-4ce8-92b4-80945af7ba40/volumes" Jul 10 17:36:00.848808 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:00.848774 2559 generic.go:356] "Generic (PLEG): container finished" podID="b617a4bd-63a8-42bd-a67f-7e72564ce56f" containerID="5e866a133c2c7915c7967e3f220b8670077b85e5431383d60066e369a0c981a2" exitCode=0 Jul 10 17:36:00.848967 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:00.848860 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7463-140489726517088" event={"ID":"b617a4bd-63a8-42bd-a67f-7e72564ce56f","Type":"ContainerDied","Data":"5e866a133c2c7915c7967e3f220b8670077b85e5431383d60066e369a0c981a2"} Jul 10 17:36:02.011989 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.011962 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:36:02.030429 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.030401 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7463-140489726517088_b617a4bd-63a8-42bd-a67f-7e72564ce56f/test-curl-7463-140489726517088/0.log" Jul 10 17:36:02.050393 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.050291 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7463-140489726517088"] Jul 10 17:36:02.053353 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.053329 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7463-140489726517088"] Jul 10 17:36:02.085579 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.085550 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4dmt\" (UniqueName: \"kubernetes.io/projected/b617a4bd-63a8-42bd-a67f-7e72564ce56f-kube-api-access-j4dmt\") pod \"b617a4bd-63a8-42bd-a67f-7e72564ce56f\" (UID: \"b617a4bd-63a8-42bd-a67f-7e72564ce56f\") " Jul 10 17:36:02.087729 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.087698 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b617a4bd-63a8-42bd-a67f-7e72564ce56f-kube-api-access-j4dmt" (OuterVolumeSpecName: "kube-api-access-j4dmt") pod "b617a4bd-63a8-42bd-a67f-7e72564ce56f" (UID: "b617a4bd-63a8-42bd-a67f-7e72564ce56f"). InnerVolumeSpecName "kube-api-access-j4dmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:36:02.187223 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.187120 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4dmt\" (UniqueName: \"kubernetes.io/projected/b617a4bd-63a8-42bd-a67f-7e72564ce56f-kube-api-access-j4dmt\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:36:02.283256 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.283228 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:02.399169 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.399130 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b617a4bd-63a8-42bd-a67f-7e72564ce56f" path="/var/lib/kubelet/pods/b617a4bd-63a8-42bd-a67f-7e72564ce56f/volumes" Jul 10 17:36:02.712418 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.712381 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:02.817501 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.817464 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:02.863384 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.863356 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7463-140489726517088" Jul 10 17:36:02.863384 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.863382 2559 scope.go:117] "RemoveContainer" containerID="5e866a133c2c7915c7967e3f220b8670077b85e5431383d60066e369a0c981a2" Jul 10 17:36:02.969563 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:02.969463 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:08.588316 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:08.588274 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:08.646621 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:08.646584 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:08.874837 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:08.874744 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:09.109353 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:09.109317 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:09.333984 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:09.333894 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:09.436427 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:09.436389 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:36:09.772741 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:09.772708 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:09.873355 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:09.873316 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:09.974088 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:09.974051 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:10.095913 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:10.095881 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:15.575949 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:15.575884 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:15.630453 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:15.630416 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:15.824778 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:15.824741 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk"] Jul 10 17:36:15.825005 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:15.824978 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" podUID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerName="istio-proxy" containerID="cri-o://c70703aff4759537e94d75e05eef432323cacf8813b00d117032aed45ace311e" gracePeriod=30 Jul 10 17:36:15.847117 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:15.847088 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:16.083452 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:16.083413 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:18.733898 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:18.733856 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wkk95"] Jul 10 17:36:20.968593 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:20.968559 2559 generic.go:356] "Generic (PLEG): container finished" podID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerID="c70703aff4759537e94d75e05eef432323cacf8813b00d117032aed45ace311e" exitCode=0 Jul 10 17:36:20.969003 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:20.968636 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" event={"ID":"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2","Type":"ContainerDied","Data":"c70703aff4759537e94d75e05eef432323cacf8813b00d117032aed45ace311e"} Jul 10 17:36:20.969003 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:20.968707 2559 scope.go:117] "RemoveContainer" containerID="007a0ee32f638e239d9e359f5182bc77097ce8e9eb2f387f0a19ae637452b73d" Jul 10 17:36:21.103366 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.103341 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:36:21.267814 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.267721 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwvt9\" (UniqueName: \"kubernetes.io/projected/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-kube-api-access-jwvt9\") pod \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " Jul 10 17:36:21.267814 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.267767 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-data\") pod \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " Jul 10 17:36:21.267814 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.267804 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-credential-socket\") pod \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " Jul 10 17:36:21.268085 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.267839 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-workload-socket\") pod \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " Jul 10 17:36:21.268085 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.267857 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-podinfo\") pod \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " Jul 10 17:36:21.268085 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.267917 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-envoy\") pod \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " Jul 10 17:36:21.268085 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.267940 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istiod-ca-cert\") pod \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " Jul 10 17:36:21.268085 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.267979 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-token\") pod \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " Jul 10 17:36:21.268085 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.268017 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-workload-certs\") pod \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\" (UID: \"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2\") " Jul 10 17:36:21.268378 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.268159 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" (UID: "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:36:21.268378 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.268325 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-workload-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:36:21.268378 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.268177 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" (UID: "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:36:21.268586 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.268400 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" (UID: "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:36:21.268737 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.268651 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" (UID: "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:36:21.268961 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.268932 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-data" (OuterVolumeSpecName: "istio-data") pod "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" (UID: "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:36:21.270264 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.270235 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-kube-api-access-jwvt9" (OuterVolumeSpecName: "kube-api-access-jwvt9") pod "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" (UID: "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2"). InnerVolumeSpecName "kube-api-access-jwvt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:36:21.270484 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.270464 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" (UID: "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:36:21.270695 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.270674 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-token" (OuterVolumeSpecName: "istio-token") pod "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" (UID: "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:36:21.270762 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.270706 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" (UID: "18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:36:21.369270 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.369226 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-credential-socket\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:36:21.369270 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.369262 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-podinfo\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:36:21.369270 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.369272 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-envoy\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:36:21.369270 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.369282 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istiod-ca-cert\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:36:21.369602 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.369291 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-token\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:36:21.369602 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.369298 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-workload-certs\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:36:21.369602 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.369306 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwvt9\" (UniqueName: \"kubernetes.io/projected/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-kube-api-access-jwvt9\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:36:21.369602 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.369314 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2-istio-data\") on node \"ip-10-0-141-228.ec2.internal\" DevicePath \"\"" Jul 10 17:36:21.569824 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.569742 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-v5nkm_2a41b3e8-d802-46aa-b2d1-685760fb4538/manager/0.log" Jul 10 17:36:21.982421 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.982381 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" event={"ID":"18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2","Type":"ContainerDied","Data":"21fdf3522b089991572d8b653c27cb57fb318ffb6eddd457d3cef24396f46b1d"} Jul 10 17:36:21.982421 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.982409 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk" Jul 10 17:36:21.982951 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:21.982466 2559 scope.go:117] "RemoveContainer" containerID="c70703aff4759537e94d75e05eef432323cacf8813b00d117032aed45ace311e" Jul 10 17:36:22.004649 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:22.004620 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk"] Jul 10 17:36:22.008588 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:22.008562 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-2482f8-openshift-default-565fc5ff7f-4cfrk"] Jul 10 17:36:22.045510 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:22.045466 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67764b8b78-2vvvl_0a2ff6c0-1e9c-418a-a34a-374fec422089/copy-manifests/0.log" Jul 10 17:36:22.051906 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:22.051878 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67764b8b78-2vvvl_0a2ff6c0-1e9c-418a-a34a-374fec422089/manager/0.log" Jul 10 17:36:22.398738 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:22.398705 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" path="/var/lib/kubelet/pods/18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2/volumes" Jul 10 17:36:23.602667 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:23.602634 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k_86807469-b337-4585-a3fb-e47255061a96/util/0.log" Jul 10 17:36:23.608663 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:23.608637 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k_86807469-b337-4585-a3fb-e47255061a96/pull/0.log" Jul 10 17:36:23.614635 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:23.614607 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k_86807469-b337-4585-a3fb-e47255061a96/extract/0.log" Jul 10 17:36:23.726176 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:23.726150 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg_9431c544-e4c6-4387-8808-f98aeb42ab81/util/0.log" Jul 10 17:36:23.732061 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:23.732028 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg_9431c544-e4c6-4387-8808-f98aeb42ab81/pull/0.log" Jul 10 17:36:23.737983 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:23.737952 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg_9431c544-e4c6-4387-8808-f98aeb42ab81/extract/0.log" Jul 10 17:36:23.855600 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:23.855517 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk_f300a123-c7ac-4314-84d2-5d6c02df5300/util/0.log" Jul 10 17:36:23.861415 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:23.861391 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk_f300a123-c7ac-4314-84d2-5d6c02df5300/pull/0.log" Jul 10 17:36:23.867301 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:23.867284 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk_f300a123-c7ac-4314-84d2-5d6c02df5300/extract/0.log" Jul 10 17:36:23.989186 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:23.989155 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x_5d271dec-621f-4218-86d1-1143a6f60558/util/0.log" Jul 10 17:36:23.995130 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:23.995108 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x_5d271dec-621f-4218-86d1-1143a6f60558/pull/0.log" Jul 10 17:36:24.001073 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:24.001055 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x_5d271dec-621f-4218-86d1-1143a6f60558/extract/0.log" Jul 10 17:36:24.847968 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:24.847933 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-wkk95_2d1e2a34-90a9-4d78-b0c6-fb875fdb588a/limitador/0.log" Jul 10 17:36:24.968205 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:24.968164 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-h2lfw_04084f37-a997-436d-9496-49d7f1ee8fcc/manager/0.log" Jul 10 17:36:25.324017 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:25.323926 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jl2jl_9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd/istio-proxy/0.log" Jul 10 17:36:26.019985 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:26.019932 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:26.070716 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:26.070675 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:26.775484 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:26.775435 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tlbsl_9ea9d9fa-c4c6-4986-b860-fd1aefcbac4b/main/0.log" Jul 10 17:36:27.006719 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:27.006689 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-g9hd7_942b9d59-d367-488a-98df-07ebb288bfa7/main/0.log" Jul 10 17:36:31.305632 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.305597 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dnfk7/must-gather-qdpkq"] Jul 10 17:36:31.306298 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.305990 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b617a4bd-63a8-42bd-a67f-7e72564ce56f" containerName="test-curl-7463-140489726517088" Jul 10 17:36:31.306298 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.306004 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b617a4bd-63a8-42bd-a67f-7e72564ce56f" containerName="test-curl-7463-140489726517088" Jul 10 17:36:31.306298 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.306027 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerName="istio-proxy" Jul 10 17:36:31.306298 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.306032 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerName="istio-proxy" Jul 10 17:36:31.306298 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.306038 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerName="istio-proxy" Jul 10 17:36:31.306298 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.306043 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerName="istio-proxy" Jul 10 17:36:31.306298 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.306103 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerName="istio-proxy" Jul 10 17:36:31.306298 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.306112 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="18ef38a8-e30c-434d-9f8a-7c2ddbbf1bb2" containerName="istio-proxy" Jul 10 17:36:31.306298 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.306120 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b617a4bd-63a8-42bd-a67f-7e72564ce56f" containerName="test-curl-7463-140489726517088" Jul 10 17:36:31.309539 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.309518 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnfk7/must-gather-qdpkq" Jul 10 17:36:31.312559 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.312534 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dnfk7\"/\"kube-root-ca.crt\"" Jul 10 17:36:31.313664 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.313636 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dnfk7\"/\"openshift-service-ca.crt\"" Jul 10 17:36:31.313664 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.313639 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dnfk7\"/\"default-dockercfg-gdhhc\"" Jul 10 17:36:31.318602 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.318577 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnfk7/must-gather-qdpkq"] Jul 10 17:36:31.320095 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.320072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:31.439261 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.439228 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:31.459455 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.459414 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/20f4fab4-0ef7-45e7-9b81-35b8122b24b1-must-gather-output\") pod \"must-gather-qdpkq\" (UID: \"20f4fab4-0ef7-45e7-9b81-35b8122b24b1\") " pod="openshift-must-gather-dnfk7/must-gather-qdpkq" Jul 10 17:36:31.459590 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.459482 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq4c5\" (UniqueName: \"kubernetes.io/projected/20f4fab4-0ef7-45e7-9b81-35b8122b24b1-kube-api-access-kq4c5\") pod \"must-gather-qdpkq\" (UID: \"20f4fab4-0ef7-45e7-9b81-35b8122b24b1\") " pod="openshift-must-gather-dnfk7/must-gather-qdpkq" Jul 10 17:36:31.560946 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.560840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/20f4fab4-0ef7-45e7-9b81-35b8122b24b1-must-gather-output\") pod \"must-gather-qdpkq\" (UID: \"20f4fab4-0ef7-45e7-9b81-35b8122b24b1\") " pod="openshift-must-gather-dnfk7/must-gather-qdpkq" Jul 10 17:36:31.560946 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.560901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq4c5\" (UniqueName: \"kubernetes.io/projected/20f4fab4-0ef7-45e7-9b81-35b8122b24b1-kube-api-access-kq4c5\") pod \"must-gather-qdpkq\" (UID: \"20f4fab4-0ef7-45e7-9b81-35b8122b24b1\") " pod="openshift-must-gather-dnfk7/must-gather-qdpkq" Jul 10 17:36:31.561254 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.561230 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/20f4fab4-0ef7-45e7-9b81-35b8122b24b1-must-gather-output\") pod \"must-gather-qdpkq\" (UID: \"20f4fab4-0ef7-45e7-9b81-35b8122b24b1\") " pod="openshift-must-gather-dnfk7/must-gather-qdpkq" Jul 10 17:36:31.569449 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.569421 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq4c5\" (UniqueName: \"kubernetes.io/projected/20f4fab4-0ef7-45e7-9b81-35b8122b24b1-kube-api-access-kq4c5\") pod \"must-gather-qdpkq\" (UID: \"20f4fab4-0ef7-45e7-9b81-35b8122b24b1\") " pod="openshift-must-gather-dnfk7/must-gather-qdpkq" Jul 10 17:36:31.620054 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.620009 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnfk7/must-gather-qdpkq" Jul 10 17:36:31.757666 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:31.757628 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnfk7/must-gather-qdpkq"] Jul 10 17:36:31.758948 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:36:31.758920 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20f4fab4_0ef7_45e7_9b81_35b8122b24b1.slice/crio-482f5d33aec61576a358961b45c27b909bf075e766ffc200c84ae6a8bd61ce9a WatchSource:0}: Error finding container 482f5d33aec61576a358961b45c27b909bf075e766ffc200c84ae6a8bd61ce9a: Status 404 returned error can't find the container with id 482f5d33aec61576a358961b45c27b909bf075e766ffc200c84ae6a8bd61ce9a Jul 10 17:36:32.039564 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:32.039528 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnfk7/must-gather-qdpkq" event={"ID":"20f4fab4-0ef7-45e7-9b81-35b8122b24b1","Type":"ContainerStarted","Data":"482f5d33aec61576a358961b45c27b909bf075e766ffc200c84ae6a8bd61ce9a"} Jul 10 17:36:33.049741 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:33.049693 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnfk7/must-gather-qdpkq" event={"ID":"20f4fab4-0ef7-45e7-9b81-35b8122b24b1","Type":"ContainerStarted","Data":"d7171ad9eff4fb2894d9c0b6e741568dc153a7bcb8972187af59e8cdb6652a80"} Jul 10 17:36:33.050115 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:33.049758 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnfk7/must-gather-qdpkq" event={"ID":"20f4fab4-0ef7-45e7-9b81-35b8122b24b1","Type":"ContainerStarted","Data":"08eb75132f2a32fac9b6fdfb69eea55171a7ee79be9d4352dea049075e66ce08"} Jul 10 17:36:33.068234 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:33.068167 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dnfk7/must-gather-qdpkq" podStartSLOduration=1.307957893 podStartE2EDuration="2.068145427s" totalImagesPullingTime="760.187534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:36:33.063984114 +0000 UTC m=+2533.189692526" watchObservedRunningTime="2026-07-10 17:36:33.068145427 +0000 UTC m=+2533.193853838" Jul 10 17:36:34.117364 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:34.117321 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xdv2b_2a54dfc8-2f7b-4e80-9f23-03c128feeb24/global-pull-secret-syncer/0.log" Jul 10 17:36:34.167169 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:34.167134 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" Jul 10 17:36:34.292111 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:34.292079 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-228.ec2.internal_b3a5470e748fa4893099e28e4ea31d34/haproxy/0.log" Jul 10 17:36:38.893951 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:38.893921 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k_86807469-b337-4585-a3fb-e47255061a96/extract/0.log" Jul 10 17:36:38.919247 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:38.919219 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k_86807469-b337-4585-a3fb-e47255061a96/util/0.log" Jul 10 17:36:38.946087 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:38.946060 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595p99k_86807469-b337-4585-a3fb-e47255061a96/pull/0.log" Jul 10 17:36:38.973539 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:38.973372 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg_9431c544-e4c6-4387-8808-f98aeb42ab81/extract/0.log" Jul 10 17:36:38.997936 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:38.997911 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg_9431c544-e4c6-4387-8808-f98aeb42ab81/util/0.log" Jul 10 17:36:39.022647 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:39.022613 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fprzg_9431c544-e4c6-4387-8808-f98aeb42ab81/pull/0.log" Jul 10 17:36:39.051445 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:39.051387 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk_f300a123-c7ac-4314-84d2-5d6c02df5300/extract/0.log" Jul 10 17:36:39.077512 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:39.077470 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk_f300a123-c7ac-4314-84d2-5d6c02df5300/util/0.log" Jul 10 17:36:39.104915 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:39.104867 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhxrk_f300a123-c7ac-4314-84d2-5d6c02df5300/pull/0.log" Jul 10 17:36:39.136135 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:39.136100 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x_5d271dec-621f-4218-86d1-1143a6f60558/extract/0.log" Jul 10 17:36:39.158579 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:39.158385 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x_5d271dec-621f-4218-86d1-1143a6f60558/util/0.log" Jul 10 17:36:39.183228 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:39.183171 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15886x_5d271dec-621f-4218-86d1-1143a6f60558/pull/0.log" Jul 10 17:36:39.768387 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:39.768357 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-wkk95_2d1e2a34-90a9-4d78-b0c6-fb875fdb588a/limitador/0.log" Jul 10 17:36:39.852448 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:39.852187 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-h2lfw_04084f37-a997-436d-9496-49d7f1ee8fcc/manager/0.log" Jul 10 17:36:41.561095 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.561068 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a09795d-9e14-4d7d-a9a9-60b272157376/alertmanager/0.log" Jul 10 17:36:41.585566 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.585533 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a09795d-9e14-4d7d-a9a9-60b272157376/config-reloader/0.log" Jul 10 17:36:41.609762 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.609728 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a09795d-9e14-4d7d-a9a9-60b272157376/kube-rbac-proxy-web/0.log" Jul 10 17:36:41.643037 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.643004 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a09795d-9e14-4d7d-a9a9-60b272157376/kube-rbac-proxy/0.log" Jul 10 17:36:41.667831 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.667748 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a09795d-9e14-4d7d-a9a9-60b272157376/kube-rbac-proxy-metric/0.log" Jul 10 17:36:41.691920 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.691887 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a09795d-9e14-4d7d-a9a9-60b272157376/prom-label-proxy/0.log" Jul 10 17:36:41.714576 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.714546 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a09795d-9e14-4d7d-a9a9-60b272157376/init-config-reloader/0.log" Jul 10 17:36:41.784565 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.784519 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-jftrl_c71664a0-87cd-4b6b-96b6-88aff6ced117/kube-state-metrics/0.log" Jul 10 17:36:41.804738 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.804707 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-jftrl_c71664a0-87cd-4b6b-96b6-88aff6ced117/kube-rbac-proxy-main/0.log" Jul 10 17:36:41.829051 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.829013 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-jftrl_c71664a0-87cd-4b6b-96b6-88aff6ced117/kube-rbac-proxy-self/0.log" Jul 10 17:36:41.893548 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.893487 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-mcnck_a52675af-70bc-428c-9efc-ee6308b051f4/monitoring-plugin/0.log" Jul 10 17:36:41.942978 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.942883 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prsg2_8e1bdb18-9cb3-4a14-9368-35be0c45a25d/node-exporter/0.log" Jul 10 17:36:41.960385 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.960347 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prsg2_8e1bdb18-9cb3-4a14-9368-35be0c45a25d/kube-rbac-proxy/0.log" Jul 10 17:36:41.987755 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:41.987720 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prsg2_8e1bdb18-9cb3-4a14-9368-35be0c45a25d/init-textfile/0.log" Jul 10 17:36:42.139541 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:42.135773 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tdzm_75c5040f-8551-4491-8725-e0b4e51a058a/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:42.178475 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:42.178443 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-z75w6_e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0/kube-rbac-proxy-main/0.log" Jul 10 17:36:42.200918 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:42.200818 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-z75w6_e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0/kube-rbac-proxy-self/0.log" Jul 10 17:36:42.222981 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:42.222935 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-z75w6_e151fc47-6131-4a64-b0ff-6fd1f0fc4ee0/openshift-state-metrics/0.log" Jul 10 17:36:42.488002 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:42.487905 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-9dbgk_103306b3-c175-4203-9469-d180393e8f69/prometheus-operator-admission-webhook/0.log" Jul 10 17:36:42.603375 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:42.603339 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b97567679-7mgmg_091039e8-bb5e-4889-8319-7c416fe7a79d/thanos-query/0.log" Jul 10 17:36:42.624837 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:42.624804 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b97567679-7mgmg_091039e8-bb5e-4889-8319-7c416fe7a79d/kube-rbac-proxy-web/0.log" Jul 10 17:36:42.647095 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:42.647063 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b97567679-7mgmg_091039e8-bb5e-4889-8319-7c416fe7a79d/kube-rbac-proxy/0.log" Jul 10 17:36:42.670853 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:42.670811 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b97567679-7mgmg_091039e8-bb5e-4889-8319-7c416fe7a79d/prom-label-proxy/0.log" Jul 10 17:36:42.693895 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:42.693851 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b97567679-7mgmg_091039e8-bb5e-4889-8319-7c416fe7a79d/kube-rbac-proxy-rules/0.log" Jul 10 17:36:42.718401 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:42.718301 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b97567679-7mgmg_091039e8-bb5e-4889-8319-7c416fe7a79d/kube-rbac-proxy-metrics/0.log" Jul 10 17:36:44.686397 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.686356 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj"] Jul 10 17:36:44.694452 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.694422 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.699480 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.699442 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj"] Jul 10 17:36:44.800711 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.800679 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-proc\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.800711 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.800714 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-lib-modules\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.800974 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.800734 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-sys\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.800974 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.800792 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-podres\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.800974 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.800842 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt6dd\" (UniqueName: \"kubernetes.io/projected/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-kube-api-access-tt6dd\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.901427 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.901381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt6dd\" (UniqueName: \"kubernetes.io/projected/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-kube-api-access-tt6dd\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.901904 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.901873 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-proc\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.902022 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.901914 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-lib-modules\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.902022 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.901943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-sys\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.902022 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.901978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-podres\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.902022 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.901991 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-proc\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.902220 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.902116 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-podres\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.902220 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.902115 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-sys\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.902220 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.902131 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-lib-modules\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:44.909957 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:44.909926 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt6dd\" (UniqueName: \"kubernetes.io/projected/0cec93f1-2936-4d31-95ca-3e2aa7b62fc7-kube-api-access-tt6dd\") pod \"perf-node-gather-daemonset-vh5bj\" (UID: \"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7\") " pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:45.010385 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:45.010281 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:45.044648 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:45.044615 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-2xv2r_4aad8b73-7fd5-4f1d-89f8-ec9bdb1afb4d/download-server/0.log" Jul 10 17:36:45.203641 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:45.203610 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj"] Jul 10 17:36:45.205176 ip-10-0-141-228 kubenswrapper[2559]: W0710 17:36:45.205139 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0cec93f1_2936_4d31_95ca_3e2aa7b62fc7.slice/crio-0dc8f25ea2912f21674972a13d8e142f5c9cb705adcccab6931308f5e17ad1c2 WatchSource:0}: Error finding container 0dc8f25ea2912f21674972a13d8e142f5c9cb705adcccab6931308f5e17ad1c2: Status 404 returned error can't find the container with id 0dc8f25ea2912f21674972a13d8e142f5c9cb705adcccab6931308f5e17ad1c2 Jul 10 17:36:46.167946 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:46.167913 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" event={"ID":"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7","Type":"ContainerStarted","Data":"f5bcc0dd8a7d73eaa8e02c210a72bf5ebdb71399bd73a1c12b1dd64e72931be5"} Jul 10 17:36:46.168457 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:46.167973 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" event={"ID":"0cec93f1-2936-4d31-95ca-3e2aa7b62fc7","Type":"ContainerStarted","Data":"0dc8f25ea2912f21674972a13d8e142f5c9cb705adcccab6931308f5e17ad1c2"} Jul 10 17:36:46.168457 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:46.168002 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:46.186845 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:46.186780 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" podStartSLOduration=2.186754896 podStartE2EDuration="2.186754896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:36:46.183188379 +0000 UTC m=+2546.308896803" watchObservedRunningTime="2026-07-10 17:36:46.186754896 +0000 UTC m=+2546.312463309" Jul 10 17:36:46.705905 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:46.705875 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmdzg_f0d79cf9-d7b1-4e10-ab4e-59c351659657/dns/0.log" Jul 10 17:36:46.728635 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:46.728588 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmdzg_f0d79cf9-d7b1-4e10-ab4e-59c351659657/kube-rbac-proxy/0.log" Jul 10 17:36:46.843929 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:46.843899 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vxt8w_b7c2a0f7-7fe0-4084-9b87-6c1e95422355/dns-node-resolver/0.log" Jul 10 17:36:47.399910 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:47.399882 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-78d8d8b7f5-btncf_9281e2bc-2561-4859-8421-26a5a3579ae9/registry/0.log" Jul 10 17:36:47.423063 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:47.423027 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bsmqb_4346a8a8-2c9c-4d06-bb87-a5364485b29d/node-ca/0.log" Jul 10 17:36:48.392958 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:48.392923 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jl2jl_9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd/istio-proxy/0.log" Jul 10 17:36:49.411378 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:49.411347 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-62nvr_31fb5d73-6dcf-4524-b6d8-cefa582aa432/serve-healthcheck-canary/0.log" Jul 10 17:36:49.947508 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:49.947461 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b4frc_de9e01f8-0325-4387-a668-8dc75d3e13bc/kube-rbac-proxy/0.log" Jul 10 17:36:49.969299 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:49.969267 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b4frc_de9e01f8-0325-4387-a668-8dc75d3e13bc/exporter/0.log" Jul 10 17:36:49.992624 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:49.992595 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b4frc_de9e01f8-0325-4387-a668-8dc75d3e13bc/extractor/0.log" Jul 10 17:36:52.189068 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:52.189034 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dnfk7/perf-node-gather-daemonset-vh5bj" Jul 10 17:36:52.405875 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:52.403535 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-v5nkm_2a41b3e8-d802-46aa-b2d1-685760fb4538/manager/0.log" Jul 10 17:36:52.606134 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:52.606105 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67764b8b78-2vvvl_0a2ff6c0-1e9c-418a-a34a-374fec422089/manager/0.log" Jul 10 17:36:52.626032 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:52.626001 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67764b8b78-2vvvl_0a2ff6c0-1e9c-418a-a34a-374fec422089/copy-manifests/0.log" Jul 10 17:36:54.113583 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:54.113542 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7bcc59c859-kfv6s_019f277a-59b3-43f3-b73c-f049d9e812b3/manager/0.log" Jul 10 17:36:54.142025 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:54.141989 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fmdg7_1afe131e-7370-4941-9bcd-377e4ead5488/openshift-lws-operator/0.log" Jul 10 17:36:55.119394 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:55.119360 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jl2jl_9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd/istio-proxy/0.log" Jul 10 17:36:56.639599 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:56.639573 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jl2jl_9f1e3bab-dae9-4a2c-a6e9-a428cf3fc6bd/istio-proxy/0.log" Jul 10 17:36:59.326323 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:59.326291 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-7bsxw_c9c041b9-a688-4931-9b40-7d9baf3a3254/migrator/0.log" Jul 10 17:36:59.345356 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:36:59.345317 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-7bsxw_c9c041b9-a688-4931-9b40-7d9baf3a3254/graceful-termination/0.log" Jul 10 17:37:01.022813 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:37:01.022788 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxchh_1cf9f299-96ec-4148-bd09-9641e7d859f1/kube-multus-additional-cni-plugins/0.log" Jul 10 17:37:01.042624 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:37:01.042592 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxchh_1cf9f299-96ec-4148-bd09-9641e7d859f1/egress-router-binary-copy/0.log" Jul 10 17:37:01.062254 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:37:01.062230 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxchh_1cf9f299-96ec-4148-bd09-9641e7d859f1/cni-plugins/0.log" Jul 10 17:37:01.083051 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:37:01.083009 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxchh_1cf9f299-96ec-4148-bd09-9641e7d859f1/bond-cni-plugin/0.log" Jul 10 17:37:01.103969 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:37:01.103927 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxchh_1cf9f299-96ec-4148-bd09-9641e7d859f1/routeoverride-cni/0.log" Jul 10 17:37:01.125453 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:37:01.125425 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxchh_1cf9f299-96ec-4148-bd09-9641e7d859f1/whereabouts-cni-bincopy/0.log" Jul 10 17:37:01.144859 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:37:01.144830 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxchh_1cf9f299-96ec-4148-bd09-9641e7d859f1/whereabouts-cni/0.log" Jul 10 17:37:01.215312 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:37:01.215283 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-q97mw_e53d41a0-f680-4189-a2d1-0500d81ca3a9/kube-multus/0.log" Jul 10 17:37:01.326188 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:37:01.326106 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wg27k_2e1b9d64-bc6b-4a12-86da-7cb47800704b/network-metrics-daemon/0.log" Jul 10 17:37:01.345133 ip-10-0-141-228 kubenswrapper[2559]: I0710 17:37:01.345092 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wg27k_2e1b9d64-bc6b-4a12-86da-7cb47800704b/kube-rbac-proxy/0.log"