Jul 22 19:59:35.153207 ip-10-0-143-196 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 19:59:35.153216 ip-10-0-143-196 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 19:59:35.153224 ip-10-0-143-196 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 19:59:35.153474 ip-10-0-143-196 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 19:59:45.225922 ip-10-0-143-196 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 19:59:45.225946 ip-10-0-143-196 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 465ed95444144049a0b45f533664d686 -- Jul 22 20:01:39.377288 ip-10-0-143-196 systemd[1]: Starting Kubernetes Kubelet... Jul 22 20:01:39.765052 ip-10-0-143-196 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 20:01:39.765052 ip-10-0-143-196 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 20:01:39.765052 ip-10-0-143-196 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 20:01:39.765052 ip-10-0-143-196 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 20:01:39.765052 ip-10-0-143-196 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 20:01:39.766136 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.765960 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 20:01:39.771168 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771148 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 20:01:39.771168 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771167 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771172 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771176 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771180 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771183 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771186 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771191 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771194 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771206 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771210 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771213 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771216 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771219 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771222 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771224 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771227 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771230 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771233 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771236 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771238 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 20:01:39.771241 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771241 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771246 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771251 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771255 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771258 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771261 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771264 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771267 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771269 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771272 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771275 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771277 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771280 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771283 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771286 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771288 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771291 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771294 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771298 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 20:01:39.771747 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771302 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771305 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771308 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771311 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771314 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771316 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771319 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771322 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771324 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771327 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771330 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771332 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771335 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771338 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771341 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771345 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771348 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771351 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771354 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 20:01:39.772259 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771356 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771362 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771365 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771369 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771372 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771375 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771378 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771382 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771384 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771387 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771390 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771393 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771396 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771400 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771403 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771406 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771410 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771413 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771416 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 20:01:39.772746 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771419 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771422 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771425 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771428 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771431 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771434 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771437 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771440 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771445 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771449 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771452 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771455 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771458 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771461 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771464 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771467 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771469 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771472 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771475 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771477 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 20:01:39.773252 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.771480 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772273 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772281 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772285 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772288 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772290 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772293 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772297 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772300 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772303 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772307 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772310 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772313 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772316 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772318 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772321 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772324 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772327 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772330 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 20:01:39.773772 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772333 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772336 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772339 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772342 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772346 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772349 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772352 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772354 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772357 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772360 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772363 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772366 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772368 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772371 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772375 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772377 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772380 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772383 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772386 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772389 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 20:01:39.774300 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772391 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772394 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772397 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772399 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772403 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772406 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772408 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772411 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772414 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772416 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772419 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772424 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772428 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772432 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772435 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772438 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772440 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772443 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772446 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772449 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 20:01:39.774830 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772451 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772454 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772457 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772460 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772464 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772467 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772471 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772474 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772477 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772479 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772482 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772485 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772487 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772490 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772493 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772496 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772498 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772502 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772505 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772507 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 20:01:39.775379 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772511 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772514 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772516 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772524 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772527 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772531 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772534 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772536 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772539 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772542 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772545 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772548 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772551 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772554 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772556 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772559 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772562 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772565 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772567 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772570 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 20:01:39.775927 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.772575 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773011 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773017 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773020 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773023 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773027 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773030 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773033 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773036 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773040 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773045 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773064 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773067 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773070 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773073 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773078 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773081 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773084 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773087 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 20:01:39.776468 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773090 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773093 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773095 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773098 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773101 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773105 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773107 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773110 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773113 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773116 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773119 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773122 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773124 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773127 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773131 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773136 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773140 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773144 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773148 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773152 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 20:01:39.777023 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773156 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773161 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773177 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773181 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773184 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773187 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773190 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773193 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773199 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773203 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773207 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773210 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773213 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773216 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773219 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773222 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773225 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773228 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773230 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 20:01:39.777558 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773233 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773236 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773238 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773241 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773244 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773247 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773250 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773253 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773256 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773259 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773262 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773264 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773267 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773271 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773275 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773278 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773280 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773284 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773286 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773289 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773292 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 20:01:39.778113 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773296 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773299 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773302 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773305 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773308 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773312 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773315 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773319 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773322 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773325 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773328 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773331 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773334 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773337 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773340 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773343 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773346 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773349 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773352 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 20:01:39.778687 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773355 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.773358 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773445 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773454 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773469 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773476 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773483 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773487 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773492 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773496 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773501 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773504 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773508 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773513 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773517 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773520 2575 flags.go:64] FLAG: --cgroup-root="" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773524 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773527 2575 flags.go:64] FLAG: --client-ca-file="" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773530 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773534 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773541 2575 flags.go:64] FLAG: --cluster-domain="" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773544 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773547 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773551 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 20:01:39.779191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773554 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773558 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773562 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773565 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773569 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773572 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773576 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773580 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773583 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773587 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773592 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773595 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773598 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773602 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773605 2575 flags.go:64] FLAG: --enable-server="true" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773608 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773613 2575 flags.go:64] FLAG: --event-burst="100" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773617 2575 flags.go:64] FLAG: --event-qps="50" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773620 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773624 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773627 2575 flags.go:64] FLAG: --eviction-hard="" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773633 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773636 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773639 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773642 2575 flags.go:64] FLAG: --eviction-soft="" Jul 22 20:01:39.779798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773645 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773648 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773651 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773655 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773658 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773661 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773664 2575 flags.go:64] FLAG: --feature-gates="" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773669 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773672 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773676 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773679 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773683 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773686 2575 flags.go:64] FLAG: --help="false" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773689 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-143-196.ec2.internal" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773693 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773696 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773699 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773703 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773706 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773709 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773712 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773715 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773718 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 20:01:39.780691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773721 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773725 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773728 2575 flags.go:64] FLAG: --kube-reserved="" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773731 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773734 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773739 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773743 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773746 2575 flags.go:64] FLAG: --lock-file="" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773749 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773751 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773755 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773761 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773764 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773767 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773770 2575 flags.go:64] FLAG: --logging-format="text" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773773 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773777 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773780 2575 flags.go:64] FLAG: --manifest-url="" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773783 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773788 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773792 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773802 2575 flags.go:64] FLAG: --max-pods="110" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773805 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773808 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773812 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 20:01:39.781814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773815 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773818 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773821 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773824 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773833 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773836 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773840 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773843 2575 flags.go:64] FLAG: --pod-cidr="" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773846 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773852 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773855 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773859 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773865 2575 flags.go:64] FLAG: --port="10250" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773871 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773874 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01d91e125bfa6ed97" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773878 2575 flags.go:64] FLAG: --qos-reserved="" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773881 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773884 2575 flags.go:64] FLAG: --register-node="true" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773887 2575 flags.go:64] FLAG: --register-with-taints="" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773891 2575 flags.go:64] FLAG: --registry-burst="10" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773894 2575 flags.go:64] FLAG: --registry-qps="5" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773898 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773901 2575 flags.go:64] FLAG: --reserved-memory="" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773904 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773908 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 20:01:39.782972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773911 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773914 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773918 2575 flags.go:64] FLAG: --runonce="false" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773921 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773928 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773931 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773935 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773938 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773941 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773944 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773948 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773951 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773974 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773978 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773981 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773985 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773989 2575 flags.go:64] FLAG: --system-cgroups="" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773991 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.773997 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.774000 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.774005 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.774010 2575 flags.go:64] FLAG: --tls-min-version="" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.774013 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.774016 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.774019 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 20:01:39.784376 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.774022 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.774025 2575 flags.go:64] FLAG: --v="2" Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.774030 2575 flags.go:64] FLAG: --version="false" Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.774034 2575 flags.go:64] FLAG: --vmodule="" Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.774038 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.774042 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774201 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774206 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774209 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774213 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774219 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774222 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774225 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774228 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774232 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774236 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774240 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774244 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774247 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774250 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774254 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 20:01:39.785322 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774256 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774259 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774262 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774265 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774268 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774270 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774275 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774278 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774281 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774284 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774287 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774289 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774292 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774295 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774298 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774301 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774304 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774306 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774309 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774313 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 20:01:39.786253 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774316 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774320 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774323 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774326 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774329 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774331 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774334 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774337 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774340 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774343 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774346 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774348 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774351 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774355 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774359 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774362 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774365 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774367 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774371 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 20:01:39.787078 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774374 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774377 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774380 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774384 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774388 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774391 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774394 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774398 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774400 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774403 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774406 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774408 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774411 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774414 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774418 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774421 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774424 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774426 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774429 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 20:01:39.787738 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774432 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774434 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774437 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774440 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774443 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774446 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774448 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774451 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774454 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774457 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774459 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774462 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774466 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774469 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774472 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774475 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774478 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774481 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774483 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774486 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 20:01:39.788380 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774489 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774491 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774494 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774497 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774500 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.774503 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.774512 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.782586 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.782609 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.782639 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.782650 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782742 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782749 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782755 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 20:01:39.788959 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782761 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782766 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782772 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782777 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782782 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782787 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782792 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782796 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782801 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782805 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782810 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782815 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782819 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782824 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782828 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782833 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782837 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782842 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782846 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782852 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 20:01:39.789361 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782857 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782861 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782866 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782870 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782875 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782879 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782885 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782891 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782896 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782900 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782905 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782909 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782913 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782918 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782922 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782926 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782931 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782935 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782940 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 20:01:39.789961 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782945 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782951 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782958 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782963 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782967 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782972 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782976 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782983 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782989 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.782995 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783000 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783005 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783010 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783016 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783020 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783026 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783030 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783035 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783039 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 20:01:39.790530 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783045 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783069 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783074 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783078 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783082 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783086 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783090 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783094 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783099 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783103 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783107 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783111 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783116 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783120 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783125 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783129 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783133 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783137 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783142 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783145 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 20:01:39.791017 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783150 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783154 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783158 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783162 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783166 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783171 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783175 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783179 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783183 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783188 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783192 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783197 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783202 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783207 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783212 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783217 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783221 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 20:01:39.791924 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783226 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.783236 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783437 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783447 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783452 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783458 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783463 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783468 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783473 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783478 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783482 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783487 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783492 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783496 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783502 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 20:01:39.792522 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783507 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783512 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783516 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783520 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783525 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783529 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783534 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783538 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783542 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783548 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783556 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783561 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783567 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783572 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783577 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783582 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783587 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783591 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783596 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783600 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 20:01:39.792921 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783605 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783612 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783618 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783624 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783629 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783634 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783639 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783644 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783648 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783653 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783658 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783662 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783669 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783675 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783680 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783685 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783690 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783695 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783700 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 20:01:39.793485 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783704 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783708 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783712 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783717 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783721 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783725 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783730 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783735 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783740 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783747 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783751 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783756 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783760 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783765 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783769 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783774 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783778 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783782 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783787 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783792 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 20:01:39.794101 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783796 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783801 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783805 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783810 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783814 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783818 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783823 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783827 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783832 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783836 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783840 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783844 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783848 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783853 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783857 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783861 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783866 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783871 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783876 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783881 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 20:01:39.794985 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783886 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 20:01:39.795794 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783891 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 20:01:39.795794 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783897 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 20:01:39.795794 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783902 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 20:01:39.795794 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783907 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 20:01:39.795794 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783911 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 20:01:39.795794 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:39.783916 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 20:01:39.795794 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.783924 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 20:01:39.795794 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.784137 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 20:01:39.795794 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.787487 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 20:01:39.795794 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.788853 2575 server.go:1019] "Starting client certificate rotation" Jul 22 20:01:39.795794 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.788959 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 20:01:39.795794 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.789011 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 20:01:39.795794 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.791773 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 20:01:39.796225 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.795592 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 20:01:39.808061 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.808031 2575 log.go:25] "Validated CRI v1 runtime API" Jul 22 20:01:39.810172 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.810148 2575 log.go:25] "Validated CRI v1 image API" Jul 22 20:01:39.811036 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.811017 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 20:01:39.816203 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.816168 2575 fs.go:135] Filesystem UUIDs: map[17a6b077-7a4f-4d88-be03-e142f29dac9f:/dev/nvme0n1p3 44328c8e-c9fa-41e9-bce7-b1c72c414ebe:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 22 20:01:39.816302 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.816201 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 20:01:39.819173 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.819155 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 20:01:39.822565 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.822429 2575 manager.go:216] Machine: {Timestamp:2026-07-22 20:01:39.821880211 +0000 UTC m=+0.334759910 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095225 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec232860e10892268230944f657aa43b SystemUUID:ec232860-e108-9226-8230-944f657aa43b BootID:465ed954-4414-4049-a0b4-5f533664d686 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7f:63:19:37:17 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7f:63:19:37:17 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:bf:c7:04:7d:74 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 20:01:39.822565 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.822559 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 20:01:39.822684 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.822657 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 20:01:39.825515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.825476 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 20:01:39.825686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.825524 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-196.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 20:01:39.826381 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.826369 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 20:01:39.826428 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.826383 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 20:01:39.826428 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.826392 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 20:01:39.826428 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.826403 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 20:01:39.827675 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.827664 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 20:01:39.830266 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.830253 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 22 20:01:39.830509 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.830498 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 20:01:39.830552 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.830540 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 22 20:01:39.830586 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.830557 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 20:01:39.830586 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.830570 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 20:01:39.830645 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.830606 2575 kubelet.go:398] "Adding apiserver pod source" Jul 22 20:01:39.830645 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.830628 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 20:01:39.831138 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.831123 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 20:01:39.831188 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.831154 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 20:01:39.834082 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.834045 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 20:01:39.838357 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.838328 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 20:01:39.838464 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.838378 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 20:01:39.839728 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839708 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 20:01:39.839728 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839731 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 20:01:39.839859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839738 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 20:01:39.839859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839744 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 20:01:39.839859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839753 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 20:01:39.839859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839762 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 20:01:39.839859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839771 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 20:01:39.839859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839777 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 20:01:39.839859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839785 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 20:01:39.839859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839791 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 20:01:39.839859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839800 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 20:01:39.839859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839806 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 20:01:39.839859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839833 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 20:01:39.839859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.839848 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 20:01:39.841228 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:39.841203 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 20:01:39.841289 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:39.841209 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 20:01:39.843309 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.843293 2575 server.go:1268] "Started kubelet" Jul 22 20:01:39.843460 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.843387 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 20:01:39.843555 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.843478 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 20:01:39.843736 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.843632 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 20:01:39.844301 ip-10-0-143-196 systemd[1]: Started Kubernetes Kubelet. Jul 22 20:01:39.845872 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.845859 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 22 20:01:39.847600 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.847583 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 20:01:39.850799 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.850779 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2s2c2" Jul 22 20:01:39.851049 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.851024 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-196.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 20:01:39.851281 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:39.851261 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 20:01:39.851446 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:39.850269 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-196.ec2.internal.18c4b51fa3dbc95c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-196.ec2.internal,UID:ip-10-0-143-196.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-196.ec2.internal,},FirstTimestamp:2026-07-22 20:01:39.843262812 +0000 UTC m=+0.356142513,LastTimestamp:2026-07-22 20:01:39.843262812 +0000 UTC m=+0.356142513,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-196.ec2.internal,}" Jul 22 20:01:39.855704 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.855682 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 20:01:39.855704 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.855694 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 20:01:39.855933 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:39.855911 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-196.ec2.internal\" not found" Jul 22 20:01:39.856760 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.856741 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 20:01:39.856839 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.856772 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 20:01:39.856839 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.856741 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 20:01:39.856961 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.856906 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 22 20:01:39.856961 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.856919 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 22 20:01:39.856961 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.856938 2575 factory.go:55] Registering systemd factory Jul 22 20:01:39.856961 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.856956 2575 factory.go:223] Registration of the systemd container factory successfully Jul 22 20:01:39.857200 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.857188 2575 factory.go:153] Registering CRI-O factory Jul 22 20:01:39.857259 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.857203 2575 factory.go:223] Registration of the crio container factory successfully Jul 22 20:01:39.857259 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.857249 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 20:01:39.857363 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.857271 2575 factory.go:103] Registering Raw factory Jul 22 20:01:39.857363 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.857308 2575 manager.go:1196] Started watching for new ooms in manager Jul 22 20:01:39.857836 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.857605 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2s2c2" Jul 22 20:01:39.858046 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.858028 2575 manager.go:349] Starting recovery of all containers Jul 22 20:01:39.864292 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:39.864265 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 20:01:39.864417 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:39.864359 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 20:01:39.868760 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.868736 2575 manager.go:354] Recovery completed Jul 22 20:01:39.876195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.876174 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 20:01:39.878430 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.878407 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientMemory" Jul 22 20:01:39.878519 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.878442 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 20:01:39.878519 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.878455 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientPID" Jul 22 20:01:39.879057 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.879040 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 20:01:39.879057 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.879054 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 20:01:39.879164 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.879066 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 20:01:39.879164 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.879086 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 22 20:01:39.879818 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.879804 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientMemory" Jul 22 20:01:39.879865 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.879833 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 20:01:39.879865 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.879846 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientPID" Jul 22 20:01:39.880442 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.880420 2575 policy_none.go:49] "None policy: Start" Jul 22 20:01:39.880499 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.880450 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 20:01:39.880499 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.880467 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 20:01:39.880606 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:39.880525 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-196.ec2.internal.18c4b51fa5f455dc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-196.ec2.internal,UID:ip-10-0-143-196.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-196.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-196.ec2.internal,},FirstTimestamp:2026-07-22 20:01:39.878426076 +0000 UTC m=+0.391305778,LastTimestamp:2026-07-22 20:01:39.878426076 +0000 UTC m=+0.391305778,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-196.ec2.internal,}" Jul 22 20:01:39.881476 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.881464 2575 policy_none.go:47] "Start" Jul 22 20:01:39.915716 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.915692 2575 manager.go:337] "Starting Device Plugin manager" Jul 22 20:01:39.915855 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:39.915742 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 20:01:39.915855 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.915757 2575 server.go:87] "Starting device plugin registration server" Jul 22 20:01:39.916086 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.916073 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 20:01:39.916158 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.916130 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 20:01:39.916286 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.916261 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 20:01:39.916398 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.916365 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 20:01:39.916398 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.916401 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 20:01:39.916947 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:39.916914 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 20:01:39.917029 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:39.916984 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-196.ec2.internal\" not found" Jul 22 20:01:39.999489 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:39.999392 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 20:01:40.000756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.000739 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 20:01:40.000812 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.000782 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 20:01:40.000812 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.000808 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 20:01:40.000886 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.000854 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 20:01:40.003252 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.003231 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 20:01:40.017138 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.017102 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 20:01:40.018084 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.018065 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientMemory" Jul 22 20:01:40.018171 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.018099 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 20:01:40.018171 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.018115 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientPID" Jul 22 20:01:40.018171 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.018146 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.024445 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.024428 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.024492 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.024453 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-196.ec2.internal\": node \"ip-10-0-143-196.ec2.internal\" not found" Jul 22 20:01:40.039425 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.039402 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-196.ec2.internal\" not found" Jul 22 20:01:40.101550 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.101491 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-196.ec2.internal"] Jul 22 20:01:40.101651 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.101603 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 20:01:40.103089 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.103073 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientMemory" Jul 22 20:01:40.103163 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.103104 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 20:01:40.103163 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.103117 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientPID" Jul 22 20:01:40.104572 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.104560 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 20:01:40.104703 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.104688 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.104749 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.104722 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 20:01:40.105375 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.105357 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientMemory" Jul 22 20:01:40.105471 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.105387 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 20:01:40.105471 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.105402 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientPID" Jul 22 20:01:40.105471 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.105406 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientMemory" Jul 22 20:01:40.105471 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.105428 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 20:01:40.105471 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.105438 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientPID" Jul 22 20:01:40.106466 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.106451 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.106521 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.106480 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 20:01:40.107216 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.107201 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientMemory" Jul 22 20:01:40.107277 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.107230 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 20:01:40.107277 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.107244 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeHasSufficientPID" Jul 22 20:01:40.136084 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.136060 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-196.ec2.internal\" not found" node="ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.139608 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.139578 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-196.ec2.internal\" not found" Jul 22 20:01:40.141117 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.141098 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-196.ec2.internal\" not found" node="ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.240134 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.240082 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-196.ec2.internal\" not found" Jul 22 20:01:40.257798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.257734 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f16d2aa87b61c01543e0be055f07431f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-196.ec2.internal\" (UID: \"f16d2aa87b61c01543e0be055f07431f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.257798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.257764 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89a76f54cfe80502fbc7b805a8de1eda-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal\" (UID: \"89a76f54cfe80502fbc7b805a8de1eda\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.257938 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.257824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89a76f54cfe80502fbc7b805a8de1eda-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal\" (UID: \"89a76f54cfe80502fbc7b805a8de1eda\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.340351 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.340293 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-196.ec2.internal\" not found" Jul 22 20:01:40.357944 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.357908 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89a76f54cfe80502fbc7b805a8de1eda-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal\" (UID: \"89a76f54cfe80502fbc7b805a8de1eda\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.358061 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.357955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89a76f54cfe80502fbc7b805a8de1eda-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal\" (UID: \"89a76f54cfe80502fbc7b805a8de1eda\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.358061 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.357975 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89a76f54cfe80502fbc7b805a8de1eda-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal\" (UID: \"89a76f54cfe80502fbc7b805a8de1eda\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.358061 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.358033 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89a76f54cfe80502fbc7b805a8de1eda-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal\" (UID: \"89a76f54cfe80502fbc7b805a8de1eda\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.358061 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.358044 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f16d2aa87b61c01543e0be055f07431f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-196.ec2.internal\" (UID: \"f16d2aa87b61c01543e0be055f07431f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.358202 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.358009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f16d2aa87b61c01543e0be055f07431f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-196.ec2.internal\" (UID: \"f16d2aa87b61c01543e0be055f07431f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.444778 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.444723 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-196.ec2.internal\" not found" Jul 22 20:01:40.444778 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.444778 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.444908 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.444777 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.548552 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.548448 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-196.ec2.internal\" not found" Jul 22 20:01:40.648946 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.648910 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-196.ec2.internal\" not found" Jul 22 20:01:40.649255 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.649238 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 20:01:40.656619 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.656599 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.668897 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.668878 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 20:01:40.671062 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.671038 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-196.ec2.internal" Jul 22 20:01:40.679937 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.679921 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 20:01:40.790112 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.790079 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 20:01:40.790596 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.790246 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 20:01:40.790596 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.790254 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 20:01:40.831748 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.831651 2575 apiserver.go:52] "Watching apiserver" Jul 22 20:01:40.851023 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.850991 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 20:01:40.854623 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.854281 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-zsbtd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24","openshift-dns/node-resolver-f99nc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal","openshift-multus/multus-additional-cni-plugins-jlwgk","openshift-multus/multus-b9gqf","openshift-multus/network-metrics-daemon-29ngf","kube-system/global-pull-secret-syncer-j7h4z","kube-system/konnectivity-agent-7jg4v","kube-system/kube-apiserver-proxy-ip-10-0-143-196.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rk8bn","openshift-image-registry/node-ca-5sdbv","openshift-network-diagnostics/network-check-target-6r6fq","openshift-network-operator/iptables-alerter-r2lsd"] Jul 22 20:01:40.856344 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.856261 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 20:01:40.856959 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.856937 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.858260 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.858241 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.859635 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.859614 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 20:01:40.859733 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.859651 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f99nc" Jul 22 20:01:40.859882 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.859866 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 20:01:40.860072 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.860056 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 20:01:40.860114 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.860100 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 20:01:40.860147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.860131 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 20:01:40.860210 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.860184 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9nptm\"" Jul 22 20:01:40.860420 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.860408 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 20:01:40.860869 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.860852 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:40.860869 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.860860 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-kubelet\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.860993 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.860882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-etc-openvswitch\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.860993 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.860902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-env-overrides\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.860993 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.860918 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-var-lib-openvswitch\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.860993 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.860936 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-log-socket\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.860993 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.860951 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-registration-dir\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.860993 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.860968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-run-netns\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861299 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-run-systemd\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861299 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861072 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-cni-bin\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861299 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861299 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861144 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbffb\" (UniqueName: \"kubernetes.io/projected/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-kube-api-access-wbffb\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861299 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861194 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-run-ovn-kubernetes\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861299 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861240 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-cni-netd\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861299 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-ovnkube-script-lib\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861589 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-device-dir\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.861589 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861337 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 20:01:40.861589 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-systemd-units\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861589 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861386 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-run-ovn\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861589 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-run-openvswitch\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861589 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-node-log\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861589 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861474 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-ovnkube-config\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861589 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861510 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-slash\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861589 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861541 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-ovn-node-metrics-cert\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.861589 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-etc-selinux\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.862045 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861600 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.862045 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861624 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-socket-dir\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.862045 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-sys-fs\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.862045 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861658 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kplbk\"" Jul 22 20:01:40.862045 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861667 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 20:01:40.862045 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjj9n\" (UniqueName: \"kubernetes.io/projected/193d7f53-af94-46a2-ad8b-97786c97081b-kube-api-access-cjj9n\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.862045 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.861709 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 20:01:40.862366 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.862089 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mzflp\"" Jul 22 20:01:40.862366 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.862129 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 20:01:40.862366 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.862197 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 20:01:40.863043 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.863027 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 20:01:40.863122 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.863107 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.863427 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.863411 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 20:01:40.863620 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.863603 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 20:01:40.863724 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.863708 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 20:01:40.864354 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.864338 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-svdmf\"" Jul 22 20:01:40.864438 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.864402 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:40.864495 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.864477 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:01:40.864596 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.864582 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 20:01:40.865149 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.865131 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 20:01:40.865559 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.865541 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p2jcx\"" Jul 22 20:01:40.865686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.865671 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:40.865759 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.865738 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:01:40.866956 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.866939 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7jg4v" Jul 22 20:01:40.868133 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.868114 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 20:01:40.868346 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.868330 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.869288 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.869257 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 20:01:40.869397 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.869259 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 20:01:40.869461 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.869421 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bd8jz\"" Jul 22 20:01:40.869797 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.869768 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5sdbv" Jul 22 20:01:40.870538 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.870518 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 20:01:40.870665 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.870642 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-646c2\"" Jul 22 20:01:40.870756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.870673 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 20:01:40.871320 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.871301 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:40.871409 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:40.871385 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:01:40.871470 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.871425 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 19:56:40 +0000 UTC" deadline="2028-04-27 16:34:17.785913984 +0000 UTC" Jul 22 20:01:40.871470 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.871448 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15476h32m36.914469029s" Jul 22 20:01:40.872003 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.871986 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 20:01:40.872152 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.872139 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 20:01:40.872223 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.872180 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 20:01:40.872281 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.872181 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-49gqp\"" Jul 22 20:01:40.872370 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.872352 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r2lsd" Jul 22 20:01:40.874464 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.874439 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 20:01:40.874563 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.874450 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 20:01:40.874729 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.874713 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 20:01:40.874773 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.874713 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rcp6c\"" Jul 22 20:01:40.896665 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.896643 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-66fjg" Jul 22 20:01:40.902864 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.902843 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-66fjg" Jul 22 20:01:40.958556 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.958527 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 20:01:40.961992 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.961958 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-cni-netd\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.962140 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-device-dir\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.962140 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962064 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-multus-socket-dir-parent\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.962140 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962119 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-cni-netd\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.962289 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962140 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-device-dir\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.962289 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962183 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-run-k8s-cni-cncf-io\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.962289 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962211 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-sysctl-d\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.962289 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962228 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-lib-modules\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.962289 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a108f216-facb-45e9-b456-516323826a5b-tmp\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.962483 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962291 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfl4v\" (UniqueName: \"kubernetes.io/projected/a108f216-facb-45e9-b456-516323826a5b-kube-api-access-kfl4v\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.962483 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-systemd-units\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.962483 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-systemd-units\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.962483 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-run-ovn\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.962483 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962471 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-run-ovn\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.962684 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962495 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-system-cni-dir\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.962684 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962525 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21f357d8-3cda-47c1-b91b-1bc4fcf45113-multus-daemon-config\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.962684 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/205d0975-9993-4a3e-ae60-573dda5e4d73-cni-binary-copy\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:40.962684 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-node-log\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.962684 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962587 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-ovnkube-config\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.962684 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-var-lib-cni-bin\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.962684 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-etc-kubernetes\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.962684 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-run\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.962684 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-sys\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.963168 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962704 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-node-log\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.963168 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88aa1673-501e-43a1-8e35-20b2a70950d0-tmp-dir\") pod \"node-resolver-f99nc\" (UID: \"88aa1673-501e-43a1-8e35-20b2a70950d0\") " pod="openshift-dns/node-resolver-f99nc" Jul 22 20:01:40.963168 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/67099d8a-f2c3-492f-a99f-6d81f030bc54-kubelet-config\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:40.963168 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-ovn-node-metrics-cert\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.963168 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962862 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-etc-selinux\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.963168 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.962965 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-etc-selinux\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.963168 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-run-multus-certs\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.963168 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a108f216-facb-45e9-b456-516323826a5b-etc-tuned\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.963168 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963077 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:40.963168 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/56ff2751-6545-46b9-860b-2d3d44c516e5-iptables-alerter-script\") pod \"iptables-alerter-r2lsd\" (UID: \"56ff2751-6545-46b9-860b-2d3d44c516e5\") " pod="openshift-network-operator/iptables-alerter-r2lsd" Jul 22 20:01:40.963168 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-sys-fs\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.963168 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963151 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-ovnkube-config\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-multus-cni-dir\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963168 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963220 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-sys-fs\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21f357d8-3cda-47c1-b91b-1bc4fcf45113-cni-binary-copy\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963272 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-run-netns\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963297 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-sysctl-conf\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtx92\" (UniqueName: \"kubernetes.io/projected/205d0975-9993-4a3e-ae60-573dda5e4d73-kube-api-access-dtx92\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56ff2751-6545-46b9-860b-2d3d44c516e5-host-slash\") pod \"iptables-alerter-r2lsd\" (UID: \"56ff2751-6545-46b9-860b-2d3d44c516e5\") " pod="openshift-network-operator/iptables-alerter-r2lsd" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963370 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-etc-openvswitch\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963418 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-multus-conf-dir\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/205d0975-9993-4a3e-ae60-573dda5e4d73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963466 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-etc-openvswitch\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963470 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/205d0975-9993-4a3e-ae60-573dda5e4d73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-var-lib-openvswitch\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963551 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-registration-dir\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963564 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-var-lib-openvswitch\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.963735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-systemd\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a96e7282-4793-4de4-b2a0-deb12e53fb4c-serviceca\") pod \"node-ca-5sdbv\" (UID: \"a96e7282-4793-4de4-b2a0-deb12e53fb4c\") " pod="openshift-image-registry/node-ca-5sdbv" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963645 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88aa1673-501e-43a1-8e35-20b2a70950d0-hosts-file\") pod \"node-resolver-f99nc\" (UID: \"88aa1673-501e-43a1-8e35-20b2a70950d0\") " pod="openshift-dns/node-resolver-f99nc" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-registration-dir\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-cni-bin\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963759 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963812 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-cni-bin\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-var-lib-cni-multus\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-modprobe-d\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-run-ovn-kubernetes\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963951 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-ovnkube-script-lib\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.963991 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-var-lib-kubelet\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964022 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-run-ovn-kubernetes\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964022 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvhld\" (UniqueName: \"kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld\") pod \"network-check-target-6r6fq\" (UID: \"2a4698b9-5706-4044-805e-be927c65786b\") " pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-var-lib-kubelet\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.964742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964110 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/90fd0be7-5de1-41c4-aeb0-59685a8907c1-konnectivity-ca\") pod \"konnectivity-agent-7jg4v\" (UID: \"90fd0be7-5de1-41c4-aeb0-59685a8907c1\") " pod="kube-system/konnectivity-agent-7jg4v" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-sysconfig\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964168 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4t98\" (UniqueName: \"kubernetes.io/projected/88aa1673-501e-43a1-8e35-20b2a70950d0-kube-api-access-s4t98\") pod \"node-resolver-f99nc\" (UID: \"88aa1673-501e-43a1-8e35-20b2a70950d0\") " pod="openshift-dns/node-resolver-f99nc" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964212 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/67099d8a-f2c3-492f-a99f-6d81f030bc54-dbus\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-run-openvswitch\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-os-release\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964331 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-run-openvswitch\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/205d0975-9993-4a3e-ae60-573dda5e4d73-cnibin\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964361 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/205d0975-9993-4a3e-ae60-573dda5e4d73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8knv\" (UniqueName: \"kubernetes.io/projected/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-kube-api-access-d8knv\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-slash\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964445 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-cnibin\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964460 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7twd\" (UniqueName: \"kubernetes.io/projected/56ff2751-6545-46b9-860b-2d3d44c516e5-kube-api-access-c7twd\") pod \"iptables-alerter-r2lsd\" (UID: \"56ff2751-6545-46b9-860b-2d3d44c516e5\") " pod="openshift-network-operator/iptables-alerter-r2lsd" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-ovnkube-script-lib\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-socket-dir\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.965365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964526 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-slash\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjj9n\" (UniqueName: \"kubernetes.io/projected/193d7f53-af94-46a2-ad8b-97786c97081b-kube-api-access-cjj9n\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964542 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-hostroot\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964584 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-kubelet\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-env-overrides\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964621 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-kubelet\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwbvc\" (UniqueName: \"kubernetes.io/projected/a96e7282-4793-4de4-b2a0-deb12e53fb4c-kube-api-access-dwbvc\") pod \"node-ca-5sdbv\" (UID: \"a96e7282-4793-4de4-b2a0-deb12e53fb4c\") " pod="openshift-image-registry/node-ca-5sdbv" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964644 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/193d7f53-af94-46a2-ad8b-97786c97081b-socket-dir\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/205d0975-9993-4a3e-ae60-573dda5e4d73-system-cni-dir\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/205d0975-9993-4a3e-ae60-573dda5e4d73-os-release\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-log-socket\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964733 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-kubernetes\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-run-netns\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-run-systemd\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964794 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbffb\" (UniqueName: \"kubernetes.io/projected/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-kube-api-access-wbffb\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964811 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsj6v\" (UniqueName: \"kubernetes.io/projected/21f357d8-3cda-47c1-b91b-1bc4fcf45113-kube-api-access-bsj6v\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:40.965876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964811 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-host-run-netns\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.966402 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964829 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/90fd0be7-5de1-41c4-aeb0-59685a8907c1-agent-certs\") pod \"konnectivity-agent-7jg4v\" (UID: \"90fd0be7-5de1-41c4-aeb0-59685a8907c1\") " pod="kube-system/konnectivity-agent-7jg4v" Jul 22 20:01:40.966402 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964868 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-log-socket\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.966402 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-host\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:40.966402 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964869 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-run-systemd\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.966402 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.964987 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-env-overrides\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.966402 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.965002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a96e7282-4793-4de4-b2a0-deb12e53fb4c-host\") pod \"node-ca-5sdbv\" (UID: \"a96e7282-4793-4de4-b2a0-deb12e53fb4c\") " pod="openshift-image-registry/node-ca-5sdbv" Jul 22 20:01:40.966581 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.966484 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-ovn-node-metrics-cert\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:40.986537 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.986509 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjj9n\" (UniqueName: \"kubernetes.io/projected/193d7f53-af94-46a2-ad8b-97786c97081b-kube-api-access-cjj9n\") pod \"aws-ebs-csi-driver-node-c6k24\" (UID: \"193d7f53-af94-46a2-ad8b-97786c97081b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:40.986661 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:40.986646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbffb\" (UniqueName: \"kubernetes.io/projected/ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b-kube-api-access-wbffb\") pod \"ovnkube-node-zsbtd\" (UID: \"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:41.040374 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:41.040319 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16d2aa87b61c01543e0be055f07431f.slice/crio-b8194e75dd2a618c7a4fab6cb8094ef67efa2410423f68e481f5fb8eb81c9fd4 WatchSource:0}: Error finding container b8194e75dd2a618c7a4fab6cb8094ef67efa2410423f68e481f5fb8eb81c9fd4: Status 404 returned error can't find the container with id b8194e75dd2a618c7a4fab6cb8094ef67efa2410423f68e481f5fb8eb81c9fd4 Jul 22 20:01:41.040815 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:41.040794 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a76f54cfe80502fbc7b805a8de1eda.slice/crio-3c24f199673f64420a7f12fbb9c59806935d967d51a643bfb87a8a1d04a8f4b6 WatchSource:0}: Error finding container 3c24f199673f64420a7f12fbb9c59806935d967d51a643bfb87a8a1d04a8f4b6: Status 404 returned error can't find the container with id 3c24f199673f64420a7f12fbb9c59806935d967d51a643bfb87a8a1d04a8f4b6 Jul 22 20:01:41.047366 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.047348 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 20:01:41.065507 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-systemd\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.065612 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065512 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a96e7282-4793-4de4-b2a0-deb12e53fb4c-serviceca\") pod \"node-ca-5sdbv\" (UID: \"a96e7282-4793-4de4-b2a0-deb12e53fb4c\") " pod="openshift-image-registry/node-ca-5sdbv" Jul 22 20:01:41.065612 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88aa1673-501e-43a1-8e35-20b2a70950d0-hosts-file\") pod \"node-resolver-f99nc\" (UID: \"88aa1673-501e-43a1-8e35-20b2a70950d0\") " pod="openshift-dns/node-resolver-f99nc" Jul 22 20:01:41.065612 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065551 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-var-lib-cni-multus\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.065612 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065569 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-modprobe-d\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.065612 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-var-lib-kubelet\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.065879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhld\" (UniqueName: \"kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld\") pod \"network-check-target-6r6fq\" (UID: \"2a4698b9-5706-4044-805e-be927c65786b\") " pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:41.065879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-var-lib-kubelet\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.065879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065657 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-systemd\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.065879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065661 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/90fd0be7-5de1-41c4-aeb0-59685a8907c1-konnectivity-ca\") pod \"konnectivity-agent-7jg4v\" (UID: \"90fd0be7-5de1-41c4-aeb0-59685a8907c1\") " pod="kube-system/konnectivity-agent-7jg4v" Jul 22 20:01:41.065879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065698 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-modprobe-d\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.065879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-sysconfig\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.065879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065737 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-var-lib-cni-multus\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.065879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065744 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-var-lib-kubelet\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.065879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4t98\" (UniqueName: \"kubernetes.io/projected/88aa1673-501e-43a1-8e35-20b2a70950d0-kube-api-access-s4t98\") pod \"node-resolver-f99nc\" (UID: \"88aa1673-501e-43a1-8e35-20b2a70950d0\") " pod="openshift-dns/node-resolver-f99nc" Jul 22 20:01:41.065879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065786 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-sysconfig\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.065879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065814 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/67099d8a-f2c3-492f-a99f-6d81f030bc54-dbus\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:41.065879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065845 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88aa1673-501e-43a1-8e35-20b2a70950d0-hosts-file\") pod \"node-resolver-f99nc\" (UID: \"88aa1673-501e-43a1-8e35-20b2a70950d0\") " pod="openshift-dns/node-resolver-f99nc" Jul 22 20:01:41.065879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-os-release\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065896 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/205d0975-9993-4a3e-ae60-573dda5e4d73-cnibin\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-os-release\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065915 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-var-lib-kubelet\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/205d0975-9993-4a3e-ae60-573dda5e4d73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065945 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/205d0975-9993-4a3e-ae60-573dda5e4d73-cnibin\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.065986 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/67099d8a-f2c3-492f-a99f-6d81f030bc54-dbus\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8knv\" (UniqueName: \"kubernetes.io/projected/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-kube-api-access-d8knv\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-cnibin\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066102 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7twd\" (UniqueName: \"kubernetes.io/projected/56ff2751-6545-46b9-860b-2d3d44c516e5-kube-api-access-c7twd\") pod \"iptables-alerter-r2lsd\" (UID: \"56ff2751-6545-46b9-860b-2d3d44c516e5\") " pod="openshift-network-operator/iptables-alerter-r2lsd" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.066106 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-hostroot\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwbvc\" (UniqueName: \"kubernetes.io/projected/a96e7282-4793-4de4-b2a0-deb12e53fb4c-kube-api-access-dwbvc\") pod \"node-ca-5sdbv\" (UID: \"a96e7282-4793-4de4-b2a0-deb12e53fb4c\") " pod="openshift-image-registry/node-ca-5sdbv" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066182 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/205d0975-9993-4a3e-ae60-573dda5e4d73-system-cni-dir\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/205d0975-9993-4a3e-ae60-573dda5e4d73-os-release\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066215 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-cnibin\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.066518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-kubernetes\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.066280 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs podName:d77c3889-b5fa-491e-b122-3b1c06ac4bd0 nodeName:}" failed. No retries permitted until 2026-07-22 20:01:41.566255564 +0000 UTC m=+2.079135255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs") pod "network-metrics-daemon-29ngf" (UID: "d77c3889-b5fa-491e-b122-3b1c06ac4bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066283 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a96e7282-4793-4de4-b2a0-deb12e53fb4c-serviceca\") pod \"node-ca-5sdbv\" (UID: \"a96e7282-4793-4de4-b2a0-deb12e53fb4c\") " pod="openshift-image-registry/node-ca-5sdbv" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066304 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-kubernetes\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsj6v\" (UniqueName: \"kubernetes.io/projected/21f357d8-3cda-47c1-b91b-1bc4fcf45113-kube-api-access-bsj6v\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066310 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/90fd0be7-5de1-41c4-aeb0-59685a8907c1-konnectivity-ca\") pod \"konnectivity-agent-7jg4v\" (UID: \"90fd0be7-5de1-41c4-aeb0-59685a8907c1\") " pod="kube-system/konnectivity-agent-7jg4v" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066343 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/90fd0be7-5de1-41c4-aeb0-59685a8907c1-agent-certs\") pod \"konnectivity-agent-7jg4v\" (UID: \"90fd0be7-5de1-41c4-aeb0-59685a8907c1\") " pod="kube-system/konnectivity-agent-7jg4v" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-hostroot\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066366 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-host\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a96e7282-4793-4de4-b2a0-deb12e53fb4c-host\") pod \"node-ca-5sdbv\" (UID: \"a96e7282-4793-4de4-b2a0-deb12e53fb4c\") " pod="openshift-image-registry/node-ca-5sdbv" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066414 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-host\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066448 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/205d0975-9993-4a3e-ae60-573dda5e4d73-system-cni-dir\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066468 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/205d0975-9993-4a3e-ae60-573dda5e4d73-os-release\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066667 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-multus-socket-dir-parent\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066720 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a96e7282-4793-4de4-b2a0-deb12e53fb4c-host\") pod \"node-ca-5sdbv\" (UID: \"a96e7282-4793-4de4-b2a0-deb12e53fb4c\") " pod="openshift-image-registry/node-ca-5sdbv" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-run-k8s-cni-cncf-io\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-multus-socket-dir-parent\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.067308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-sysctl-d\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-lib-modules\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066793 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-run-k8s-cni-cncf-io\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a108f216-facb-45e9-b456-516323826a5b-tmp\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfl4v\" (UniqueName: \"kubernetes.io/projected/a108f216-facb-45e9-b456-516323826a5b-kube-api-access-kfl4v\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-system-cni-dir\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21f357d8-3cda-47c1-b91b-1bc4fcf45113-multus-daemon-config\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066897 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-sysctl-d\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066907 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/205d0975-9993-4a3e-ae60-573dda5e4d73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066900 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-lib-modules\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066903 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/205d0975-9993-4a3e-ae60-573dda5e4d73-cni-binary-copy\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.066978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-var-lib-cni-bin\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067023 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-etc-kubernetes\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-run\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-sys\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88aa1673-501e-43a1-8e35-20b2a70950d0-tmp-dir\") pod \"node-resolver-f99nc\" (UID: \"88aa1673-501e-43a1-8e35-20b2a70950d0\") " pod="openshift-dns/node-resolver-f99nc" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/67099d8a-f2c3-492f-a99f-6d81f030bc54-kubelet-config\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-run-multus-certs\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.068271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067176 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a108f216-facb-45e9-b456-516323826a5b-etc-tuned\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067170 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-run\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067232 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-var-lib-cni-bin\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/56ff2751-6545-46b9-860b-2d3d44c516e5-iptables-alerter-script\") pod \"iptables-alerter-r2lsd\" (UID: \"56ff2751-6545-46b9-860b-2d3d44c516e5\") " pod="openshift-network-operator/iptables-alerter-r2lsd" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067273 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-etc-kubernetes\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-multus-cni-dir\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067323 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-run-multus-certs\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067352 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21f357d8-3cda-47c1-b91b-1bc4fcf45113-cni-binary-copy\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067356 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-multus-cni-dir\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067379 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-run-netns\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067411 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-host-run-netns\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-sysctl-conf\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067444 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtx92\" (UniqueName: \"kubernetes.io/projected/205d0975-9993-4a3e-ae60-573dda5e4d73-kube-api-access-dtx92\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067447 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/205d0975-9993-4a3e-ae60-573dda5e4d73-cni-binary-copy\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56ff2751-6545-46b9-860b-2d3d44c516e5-host-slash\") pod \"iptables-alerter-r2lsd\" (UID: \"56ff2751-6545-46b9-860b-2d3d44c516e5\") " pod="openshift-network-operator/iptables-alerter-r2lsd" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-multus-conf-dir\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067519 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-system-cni-dir\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.069195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067531 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21f357d8-3cda-47c1-b91b-1bc4fcf45113-multus-daemon-config\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/205d0975-9993-4a3e-ae60-573dda5e4d73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/205d0975-9993-4a3e-ae60-573dda5e4d73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067725 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/205d0975-9993-4a3e-ae60-573dda5e4d73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067810 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-sys\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.068133 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88aa1673-501e-43a1-8e35-20b2a70950d0-tmp-dir\") pod \"node-resolver-f99nc\" (UID: \"88aa1673-501e-43a1-8e35-20b2a70950d0\") " pod="openshift-dns/node-resolver-f99nc" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.068202 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/56ff2751-6545-46b9-860b-2d3d44c516e5-iptables-alerter-script\") pod \"iptables-alerter-r2lsd\" (UID: \"56ff2751-6545-46b9-860b-2d3d44c516e5\") " pod="openshift-network-operator/iptables-alerter-r2lsd" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.068202 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/205d0975-9993-4a3e-ae60-573dda5e4d73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.067537 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a108f216-facb-45e9-b456-516323826a5b-etc-sysctl-conf\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.068228 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/67099d8a-f2c3-492f-a99f-6d81f030bc54-kubelet-config\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.068269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21f357d8-3cda-47c1-b91b-1bc4fcf45113-multus-conf-dir\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.068344 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.068393 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret podName:67099d8a-f2c3-492f-a99f-6d81f030bc54 nodeName:}" failed. No retries permitted until 2026-07-22 20:01:41.568377281 +0000 UTC m=+2.081256977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret") pod "global-pull-secret-syncer-j7h4z" (UID: "67099d8a-f2c3-492f-a99f-6d81f030bc54") : object "kube-system"/"original-pull-secret" not registered Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.068392 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56ff2751-6545-46b9-860b-2d3d44c516e5-host-slash\") pod \"iptables-alerter-r2lsd\" (UID: \"56ff2751-6545-46b9-860b-2d3d44c516e5\") " pod="openshift-network-operator/iptables-alerter-r2lsd" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.068626 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21f357d8-3cda-47c1-b91b-1bc4fcf45113-cni-binary-copy\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.069466 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a108f216-facb-45e9-b456-516323826a5b-tmp\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.069506 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/90fd0be7-5de1-41c4-aeb0-59685a8907c1-agent-certs\") pod \"konnectivity-agent-7jg4v\" (UID: \"90fd0be7-5de1-41c4-aeb0-59685a8907c1\") " pod="kube-system/konnectivity-agent-7jg4v" Jul 22 20:01:41.069756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.069733 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a108f216-facb-45e9-b456-516323826a5b-etc-tuned\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.075502 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.075486 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 20:01:41.075557 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.075505 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 20:01:41.075557 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.075515 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hvhld for pod openshift-network-diagnostics/network-check-target-6r6fq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:01:41.075629 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.075586 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld podName:2a4698b9-5706-4044-805e-be927c65786b nodeName:}" failed. No retries permitted until 2026-07-22 20:01:41.575574075 +0000 UTC m=+2.088453800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hvhld" (UniqueName: "kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld") pod "network-check-target-6r6fq" (UID: "2a4698b9-5706-4044-805e-be927c65786b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:01:41.077225 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.077205 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8knv\" (UniqueName: \"kubernetes.io/projected/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-kube-api-access-d8knv\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:41.077928 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.077907 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4t98\" (UniqueName: \"kubernetes.io/projected/88aa1673-501e-43a1-8e35-20b2a70950d0-kube-api-access-s4t98\") pod \"node-resolver-f99nc\" (UID: \"88aa1673-501e-43a1-8e35-20b2a70950d0\") " pod="openshift-dns/node-resolver-f99nc" Jul 22 20:01:41.081504 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.081476 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtx92\" (UniqueName: \"kubernetes.io/projected/205d0975-9993-4a3e-ae60-573dda5e4d73-kube-api-access-dtx92\") pod \"multus-additional-cni-plugins-jlwgk\" (UID: \"205d0975-9993-4a3e-ae60-573dda5e4d73\") " pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.081654 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.081635 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7twd\" (UniqueName: \"kubernetes.io/projected/56ff2751-6545-46b9-860b-2d3d44c516e5-kube-api-access-c7twd\") pod \"iptables-alerter-r2lsd\" (UID: \"56ff2751-6545-46b9-860b-2d3d44c516e5\") " pod="openshift-network-operator/iptables-alerter-r2lsd" Jul 22 20:01:41.081858 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.081809 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsj6v\" (UniqueName: \"kubernetes.io/projected/21f357d8-3cda-47c1-b91b-1bc4fcf45113-kube-api-access-bsj6v\") pod \"multus-b9gqf\" (UID: \"21f357d8-3cda-47c1-b91b-1bc4fcf45113\") " pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.082118 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.082100 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfl4v\" (UniqueName: \"kubernetes.io/projected/a108f216-facb-45e9-b456-516323826a5b-kube-api-access-kfl4v\") pod \"tuned-rk8bn\" (UID: \"a108f216-facb-45e9-b456-516323826a5b\") " pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.082253 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.082238 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwbvc\" (UniqueName: \"kubernetes.io/projected/a96e7282-4793-4de4-b2a0-deb12e53fb4c-kube-api-access-dwbvc\") pod \"node-ca-5sdbv\" (UID: \"a96e7282-4793-4de4-b2a0-deb12e53fb4c\") " pod="openshift-image-registry/node-ca-5sdbv" Jul 22 20:01:41.160337 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.160309 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 20:01:41.193276 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.193222 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:01:41.203183 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:41.203151 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1c3d0f_7d3c_470e_8ef1_d20299d5d75b.slice/crio-a220c91f2d2f61b644742e97730fe4b66e4df34eb9f558326b2ba80e783d7df9 WatchSource:0}: Error finding container a220c91f2d2f61b644742e97730fe4b66e4df34eb9f558326b2ba80e783d7df9: Status 404 returned error can't find the container with id a220c91f2d2f61b644742e97730fe4b66e4df34eb9f558326b2ba80e783d7df9 Jul 22 20:01:41.210684 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.210664 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" Jul 22 20:01:41.218372 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:41.218336 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193d7f53_af94_46a2_ad8b_97786c97081b.slice/crio-a124389661a529a57ec29cdd6e0ee5013ab679157a0c6dbc9f79544197d5676d WatchSource:0}: Error finding container a124389661a529a57ec29cdd6e0ee5013ab679157a0c6dbc9f79544197d5676d: Status 404 returned error can't find the container with id a124389661a529a57ec29cdd6e0ee5013ab679157a0c6dbc9f79544197d5676d Jul 22 20:01:41.230506 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.230471 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f99nc" Jul 22 20:01:41.238389 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:41.238365 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88aa1673_501e_43a1_8e35_20b2a70950d0.slice/crio-ec5d11a10d179e3282b0c93d9ad2975cac9515f4de1c36b3a1a44a7e38d853b5 WatchSource:0}: Error finding container ec5d11a10d179e3282b0c93d9ad2975cac9515f4de1c36b3a1a44a7e38d853b5: Status 404 returned error can't find the container with id ec5d11a10d179e3282b0c93d9ad2975cac9515f4de1c36b3a1a44a7e38d853b5 Jul 22 20:01:41.240003 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.239974 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jlwgk" Jul 22 20:01:41.248013 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:41.247975 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205d0975_9993_4a3e_ae60_573dda5e4d73.slice/crio-72fe12ce350ab91035c520656254de80576f22340c44f816da4dbdd3c0f44ae4 WatchSource:0}: Error finding container 72fe12ce350ab91035c520656254de80576f22340c44f816da4dbdd3c0f44ae4: Status 404 returned error can't find the container with id 72fe12ce350ab91035c520656254de80576f22340c44f816da4dbdd3c0f44ae4 Jul 22 20:01:41.260330 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.260306 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b9gqf" Jul 22 20:01:41.267452 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:41.267427 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f357d8_3cda_47c1_b91b_1bc4fcf45113.slice/crio-42adb29f8c7d2f6069859112343732fdb132f232bb508620d7c99457fbe72582 WatchSource:0}: Error finding container 42adb29f8c7d2f6069859112343732fdb132f232bb508620d7c99457fbe72582: Status 404 returned error can't find the container with id 42adb29f8c7d2f6069859112343732fdb132f232bb508620d7c99457fbe72582 Jul 22 20:01:41.279566 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.279545 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7jg4v" Jul 22 20:01:41.287307 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:41.287281 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fd0be7_5de1_41c4_aeb0_59685a8907c1.slice/crio-55789a766d7bfa51d3d7a06c6f73ad9b4f3da1320827c7863a1b08d16d8dfa25 WatchSource:0}: Error finding container 55789a766d7bfa51d3d7a06c6f73ad9b4f3da1320827c7863a1b08d16d8dfa25: Status 404 returned error can't find the container with id 55789a766d7bfa51d3d7a06c6f73ad9b4f3da1320827c7863a1b08d16d8dfa25 Jul 22 20:01:41.295828 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.295808 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" Jul 22 20:01:41.302638 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:41.302608 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda108f216_facb_45e9_b456_516323826a5b.slice/crio-f3ffcc0e6409f483af492641dbe08fa4234839fa089ea968e0b4a88be3e55170 WatchSource:0}: Error finding container f3ffcc0e6409f483af492641dbe08fa4234839fa089ea968e0b4a88be3e55170: Status 404 returned error can't find the container with id f3ffcc0e6409f483af492641dbe08fa4234839fa089ea968e0b4a88be3e55170 Jul 22 20:01:41.308025 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.308001 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5sdbv" Jul 22 20:01:41.313772 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.313755 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r2lsd" Jul 22 20:01:41.316119 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:41.316095 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96e7282_4793_4de4_b2a0_deb12e53fb4c.slice/crio-b51840345064a88d44b2f1013d39e7b82e854c3c4a34cec2020a2a8ca0dc345e WatchSource:0}: Error finding container b51840345064a88d44b2f1013d39e7b82e854c3c4a34cec2020a2a8ca0dc345e: Status 404 returned error can't find the container with id b51840345064a88d44b2f1013d39e7b82e854c3c4a34cec2020a2a8ca0dc345e Jul 22 20:01:41.325683 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:01:41.325657 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ff2751_6545_46b9_860b_2d3d44c516e5.slice/crio-2d2cad1aed91bab7b8712a048ca02309ff654980d290cbebc7632ab64490734d WatchSource:0}: Error finding container 2d2cad1aed91bab7b8712a048ca02309ff654980d290cbebc7632ab64490734d: Status 404 returned error can't find the container with id 2d2cad1aed91bab7b8712a048ca02309ff654980d290cbebc7632ab64490734d Jul 22 20:01:41.337453 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.337391 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 20:01:41.570829 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.570792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:41.571003 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.570860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:41.571003 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.570953 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 20:01:41.571003 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.570963 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:01:41.571155 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.571035 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret podName:67099d8a-f2c3-492f-a99f-6d81f030bc54 nodeName:}" failed. No retries permitted until 2026-07-22 20:01:42.571013489 +0000 UTC m=+3.083893192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret") pod "global-pull-secret-syncer-j7h4z" (UID: "67099d8a-f2c3-492f-a99f-6d81f030bc54") : object "kube-system"/"original-pull-secret" not registered Jul 22 20:01:41.571155 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.571054 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs podName:d77c3889-b5fa-491e-b122-3b1c06ac4bd0 nodeName:}" failed. No retries permitted until 2026-07-22 20:01:42.571044289 +0000 UTC m=+3.083923975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs") pod "network-metrics-daemon-29ngf" (UID: "d77c3889-b5fa-491e-b122-3b1c06ac4bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:01:41.671834 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.671747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhld\" (UniqueName: \"kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld\") pod \"network-check-target-6r6fq\" (UID: \"2a4698b9-5706-4044-805e-be927c65786b\") " pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:41.672017 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.671919 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 20:01:41.672017 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.671945 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 20:01:41.672017 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.671960 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hvhld for pod openshift-network-diagnostics/network-check-target-6r6fq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:01:41.672198 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:41.672028 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld podName:2a4698b9-5706-4044-805e-be927c65786b nodeName:}" failed. No retries permitted until 2026-07-22 20:01:42.672005856 +0000 UTC m=+3.184885541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvhld" (UniqueName: "kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld") pod "network-check-target-6r6fq" (UID: "2a4698b9-5706-4044-805e-be927c65786b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:01:41.906942 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.906070 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 19:56:40 +0000 UTC" deadline="2028-04-11 08:48:32.554145052 +0000 UTC" Jul 22 20:01:41.906942 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:41.906130 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15084h46m50.648019391s" Jul 22 20:01:42.013580 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.013403 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f99nc" event={"ID":"88aa1673-501e-43a1-8e35-20b2a70950d0","Type":"ContainerStarted","Data":"ec5d11a10d179e3282b0c93d9ad2975cac9515f4de1c36b3a1a44a7e38d853b5"} Jul 22 20:01:42.015796 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.015681 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" event={"ID":"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b","Type":"ContainerStarted","Data":"a220c91f2d2f61b644742e97730fe4b66e4df34eb9f558326b2ba80e783d7df9"} Jul 22 20:01:42.019928 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.019901 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-196.ec2.internal" event={"ID":"f16d2aa87b61c01543e0be055f07431f","Type":"ContainerStarted","Data":"b8194e75dd2a618c7a4fab6cb8094ef67efa2410423f68e481f5fb8eb81c9fd4"} Jul 22 20:01:42.022325 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.022299 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5sdbv" event={"ID":"a96e7282-4793-4de4-b2a0-deb12e53fb4c","Type":"ContainerStarted","Data":"b51840345064a88d44b2f1013d39e7b82e854c3c4a34cec2020a2a8ca0dc345e"} Jul 22 20:01:42.024917 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.024893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlwgk" event={"ID":"205d0975-9993-4a3e-ae60-573dda5e4d73","Type":"ContainerStarted","Data":"72fe12ce350ab91035c520656254de80576f22340c44f816da4dbdd3c0f44ae4"} Jul 22 20:01:42.029787 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.029751 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" event={"ID":"193d7f53-af94-46a2-ad8b-97786c97081b","Type":"ContainerStarted","Data":"a124389661a529a57ec29cdd6e0ee5013ab679157a0c6dbc9f79544197d5676d"} Jul 22 20:01:42.033092 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.033063 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" event={"ID":"89a76f54cfe80502fbc7b805a8de1eda","Type":"ContainerStarted","Data":"3c24f199673f64420a7f12fbb9c59806935d967d51a643bfb87a8a1d04a8f4b6"} Jul 22 20:01:42.037004 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.036966 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r2lsd" event={"ID":"56ff2751-6545-46b9-860b-2d3d44c516e5","Type":"ContainerStarted","Data":"2d2cad1aed91bab7b8712a048ca02309ff654980d290cbebc7632ab64490734d"} Jul 22 20:01:42.040318 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.040288 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" event={"ID":"a108f216-facb-45e9-b456-516323826a5b","Type":"ContainerStarted","Data":"f3ffcc0e6409f483af492641dbe08fa4234839fa089ea968e0b4a88be3e55170"} Jul 22 20:01:42.042354 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.042332 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7jg4v" event={"ID":"90fd0be7-5de1-41c4-aeb0-59685a8907c1","Type":"ContainerStarted","Data":"55789a766d7bfa51d3d7a06c6f73ad9b4f3da1320827c7863a1b08d16d8dfa25"} Jul 22 20:01:42.046356 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.046318 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b9gqf" event={"ID":"21f357d8-3cda-47c1-b91b-1bc4fcf45113","Type":"ContainerStarted","Data":"42adb29f8c7d2f6069859112343732fdb132f232bb508620d7c99457fbe72582"} Jul 22 20:01:42.141799 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.141767 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 20:01:42.579873 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.579828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:42.580063 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.579905 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:42.580154 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:42.580085 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:01:42.580211 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:42.580161 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs podName:d77c3889-b5fa-491e-b122-3b1c06ac4bd0 nodeName:}" failed. No retries permitted until 2026-07-22 20:01:44.580137701 +0000 UTC m=+5.093017401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs") pod "network-metrics-daemon-29ngf" (UID: "d77c3889-b5fa-491e-b122-3b1c06ac4bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:01:42.580211 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:42.580085 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 20:01:42.580322 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:42.580248 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret podName:67099d8a-f2c3-492f-a99f-6d81f030bc54 nodeName:}" failed. No retries permitted until 2026-07-22 20:01:44.580228196 +0000 UTC m=+5.093107885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret") pod "global-pull-secret-syncer-j7h4z" (UID: "67099d8a-f2c3-492f-a99f-6d81f030bc54") : object "kube-system"/"original-pull-secret" not registered Jul 22 20:01:42.644497 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.644465 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 20:01:42.680348 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.680280 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhld\" (UniqueName: \"kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld\") pod \"network-check-target-6r6fq\" (UID: \"2a4698b9-5706-4044-805e-be927c65786b\") " pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:42.680523 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:42.680489 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 20:01:42.680523 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:42.680511 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 20:01:42.680523 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:42.680523 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hvhld for pod openshift-network-diagnostics/network-check-target-6r6fq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:01:42.680682 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:42.680591 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld podName:2a4698b9-5706-4044-805e-be927c65786b nodeName:}" failed. No retries permitted until 2026-07-22 20:01:44.680564982 +0000 UTC m=+5.193444675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvhld" (UniqueName: "kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld") pod "network-check-target-6r6fq" (UID: "2a4698b9-5706-4044-805e-be927c65786b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:01:42.906852 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.906764 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 19:56:40 +0000 UTC" deadline="2028-01-08 06:24:30.511134378 +0000 UTC" Jul 22 20:01:42.906852 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:42.906804 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12826h22m47.604334113s" Jul 22 20:01:43.001397 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:43.001366 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:43.001584 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:43.001505 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:01:43.001932 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:43.001911 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:43.002058 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:43.002038 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:01:43.002135 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:43.002118 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:43.002235 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:43.002208 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:01:44.059827 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:44.059575 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" event={"ID":"89a76f54cfe80502fbc7b805a8de1eda","Type":"ContainerStarted","Data":"82fb8f895b8ceecb1b88ef816e2d1f4db632a1258b1eb4e4db2e9f203e68b745"} Jul 22 20:01:44.598306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:44.597969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:44.598306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:44.598080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:44.598306 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:44.598141 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 20:01:44.598306 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:44.598217 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:01:44.598306 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:44.598224 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret podName:67099d8a-f2c3-492f-a99f-6d81f030bc54 nodeName:}" failed. No retries permitted until 2026-07-22 20:01:48.598202174 +0000 UTC m=+9.111081865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret") pod "global-pull-secret-syncer-j7h4z" (UID: "67099d8a-f2c3-492f-a99f-6d81f030bc54") : object "kube-system"/"original-pull-secret" not registered Jul 22 20:01:44.598306 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:44.598284 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs podName:d77c3889-b5fa-491e-b122-3b1c06ac4bd0 nodeName:}" failed. No retries permitted until 2026-07-22 20:01:48.598266494 +0000 UTC m=+9.111146177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs") pod "network-metrics-daemon-29ngf" (UID: "d77c3889-b5fa-491e-b122-3b1c06ac4bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:01:44.699205 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:44.699166 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhld\" (UniqueName: \"kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld\") pod \"network-check-target-6r6fq\" (UID: \"2a4698b9-5706-4044-805e-be927c65786b\") " pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:44.699414 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:44.699393 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 20:01:44.699475 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:44.699422 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 20:01:44.699475 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:44.699436 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hvhld for pod openshift-network-diagnostics/network-check-target-6r6fq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:01:44.699638 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:44.699493 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld podName:2a4698b9-5706-4044-805e-be927c65786b nodeName:}" failed. No retries permitted until 2026-07-22 20:01:48.699477951 +0000 UTC m=+9.212357648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvhld" (UniqueName: "kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld") pod "network-check-target-6r6fq" (UID: "2a4698b9-5706-4044-805e-be927c65786b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:01:45.002039 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:45.002008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:45.002039 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:45.002025 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:45.002310 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:45.002125 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:45.002310 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:45.002156 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:01:45.002310 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:45.002212 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:01:45.002310 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:45.002296 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:01:45.064423 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:45.064384 2575 generic.go:356] "Generic (PLEG): container finished" podID="89a76f54cfe80502fbc7b805a8de1eda" containerID="82fb8f895b8ceecb1b88ef816e2d1f4db632a1258b1eb4e4db2e9f203e68b745" exitCode=0 Jul 22 20:01:45.064872 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:45.064437 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" event={"ID":"89a76f54cfe80502fbc7b805a8de1eda","Type":"ContainerDied","Data":"82fb8f895b8ceecb1b88ef816e2d1f4db632a1258b1eb4e4db2e9f203e68b745"} Jul 22 20:01:47.001695 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:47.001561 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:47.001695 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:47.001561 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:47.001695 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:47.001573 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:47.002290 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:47.001782 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:01:47.002290 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:47.001904 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:01:47.002290 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:47.002017 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:01:48.634573 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:48.634485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:48.635027 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:48.634580 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:48.635027 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:48.634661 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:01:48.635027 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:48.634690 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 20:01:48.635027 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:48.634739 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs podName:d77c3889-b5fa-491e-b122-3b1c06ac4bd0 nodeName:}" failed. No retries permitted until 2026-07-22 20:01:56.634717127 +0000 UTC m=+17.147596812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs") pod "network-metrics-daemon-29ngf" (UID: "d77c3889-b5fa-491e-b122-3b1c06ac4bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:01:48.635027 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:48.634759 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret podName:67099d8a-f2c3-492f-a99f-6d81f030bc54 nodeName:}" failed. No retries permitted until 2026-07-22 20:01:56.63474998 +0000 UTC m=+17.147629670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret") pod "global-pull-secret-syncer-j7h4z" (UID: "67099d8a-f2c3-492f-a99f-6d81f030bc54") : object "kube-system"/"original-pull-secret" not registered Jul 22 20:01:48.735576 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:48.735009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhld\" (UniqueName: \"kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld\") pod \"network-check-target-6r6fq\" (UID: \"2a4698b9-5706-4044-805e-be927c65786b\") " pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:48.735576 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:48.735164 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 20:01:48.735576 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:48.735182 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 20:01:48.735576 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:48.735194 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hvhld for pod openshift-network-diagnostics/network-check-target-6r6fq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:01:48.735576 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:48.735246 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld podName:2a4698b9-5706-4044-805e-be927c65786b nodeName:}" failed. No retries permitted until 2026-07-22 20:01:56.735228679 +0000 UTC m=+17.248108387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvhld" (UniqueName: "kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld") pod "network-check-target-6r6fq" (UID: "2a4698b9-5706-4044-805e-be927c65786b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:01:49.001926 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:49.001312 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:49.001926 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:49.001440 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:01:49.001926 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:49.001740 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:49.001926 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:49.001870 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:01:49.002250 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:49.001939 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:49.002250 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:49.002059 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:01:51.001835 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:51.001796 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:51.002312 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:51.001797 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:51.002312 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:51.001797 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:51.002312 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:51.001922 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:01:51.002312 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:51.002034 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:01:51.002312 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:51.002170 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:01:53.001225 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:53.001188 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:53.001225 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:53.001209 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:53.001225 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:53.001188 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:53.001807 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:53.001332 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:01:53.001807 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:53.001403 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:01:53.001807 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:53.001473 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:01:55.001962 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:55.001926 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:55.001962 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:55.001947 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:55.001962 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:55.001971 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:55.002471 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:55.002099 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:01:55.002471 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:55.002203 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:01:55.002471 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:55.002288 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:01:56.691338 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:56.691307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:56.691702 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:56.691367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:56.691702 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:56.691466 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 20:01:56.691702 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:56.691471 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:01:56.691702 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:56.691534 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret podName:67099d8a-f2c3-492f-a99f-6d81f030bc54 nodeName:}" failed. No retries permitted until 2026-07-22 20:02:12.691517266 +0000 UTC m=+33.204396973 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret") pod "global-pull-secret-syncer-j7h4z" (UID: "67099d8a-f2c3-492f-a99f-6d81f030bc54") : object "kube-system"/"original-pull-secret" not registered Jul 22 20:01:56.691702 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:56.691551 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs podName:d77c3889-b5fa-491e-b122-3b1c06ac4bd0 nodeName:}" failed. No retries permitted until 2026-07-22 20:02:12.69154451 +0000 UTC m=+33.204424194 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs") pod "network-metrics-daemon-29ngf" (UID: "d77c3889-b5fa-491e-b122-3b1c06ac4bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:01:56.792741 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:56.792705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhld\" (UniqueName: \"kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld\") pod \"network-check-target-6r6fq\" (UID: \"2a4698b9-5706-4044-805e-be927c65786b\") " pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:56.792946 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:56.792889 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 20:01:56.792946 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:56.792916 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 20:01:56.792946 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:56.792929 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hvhld for pod openshift-network-diagnostics/network-check-target-6r6fq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:01:56.793125 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:56.793001 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld podName:2a4698b9-5706-4044-805e-be927c65786b nodeName:}" failed. No retries permitted until 2026-07-22 20:02:12.792965243 +0000 UTC m=+33.305844928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvhld" (UniqueName: "kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld") pod "network-check-target-6r6fq" (UID: "2a4698b9-5706-4044-805e-be927c65786b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:01:57.001842 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:57.001763 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:57.002045 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:57.001764 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:57.002045 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:57.001895 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:01:57.002045 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:57.001777 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:57.002045 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:57.002028 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:01:57.002202 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:57.002080 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:01:59.002036 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:59.001995 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:01:59.002534 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:59.002129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:01:59.002534 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:59.002140 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:01:59.002534 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:01:59.002159 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:01:59.002534 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:59.002283 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:01:59.002534 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:01:59.002388 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:02:01.001576 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:01.001551 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:02:01.001941 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:01.001679 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:02:01.001941 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:01.001771 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:02:01.001941 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:01.001825 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:02:01.001941 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:01.001919 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:02:01.002166 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:01.002027 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:02:02.109182 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.108752 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f99nc" event={"ID":"88aa1673-501e-43a1-8e35-20b2a70950d0","Type":"ContainerStarted","Data":"a5ee7b701ee476879c2a2c0aef4790601fa4eb817f6de1708dabe1b647e01464"} Jul 22 20:02:02.112160 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.112131 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:02:02.112505 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.112483 2575 generic.go:356] "Generic (PLEG): container finished" podID="ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b" containerID="7d0e406d043d3ea9050baea421000895345c3aaf28ade29982dd85fca2265030" exitCode=1 Jul 22 20:02:02.112596 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.112550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" event={"ID":"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b","Type":"ContainerStarted","Data":"79f283c79ad056ed88b33aa952fa10970f09832ed04e2ac719faac960347540a"} Jul 22 20:02:02.112647 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.112625 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" event={"ID":"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b","Type":"ContainerStarted","Data":"9310eae19aab2509ef0727771f6a129afc01f58ab70f9113db033478422d0b61"} Jul 22 20:02:02.112685 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.112674 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" event={"ID":"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b","Type":"ContainerStarted","Data":"17cda1a1f103441b6f11b62cfa2394ff11836ff85b539ab98d8f5febb4ce5bce"} Jul 22 20:02:02.112731 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.112722 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" event={"ID":"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b","Type":"ContainerStarted","Data":"a3ca05cf20e7f37862590e52eb7c6999d8a404aeb412e61cba7cc23f6d53fa04"} Jul 22 20:02:02.112810 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.112767 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" event={"ID":"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b","Type":"ContainerDied","Data":"7d0e406d043d3ea9050baea421000895345c3aaf28ade29982dd85fca2265030"} Jul 22 20:02:02.112994 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.112844 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" event={"ID":"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b","Type":"ContainerStarted","Data":"f41825d660c9fa651fdebe5a37609ba8891c4a0c2424c5d4a61d4ddaf4cb82a3"} Jul 22 20:02:02.114365 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.114338 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-196.ec2.internal" event={"ID":"f16d2aa87b61c01543e0be055f07431f","Type":"ContainerStarted","Data":"5b693bbf5fc1112139a38d4eb5e5eba02aa264f44d1ee560aa9dcbd2cc824dad"} Jul 22 20:02:02.116025 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.116003 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5sdbv" event={"ID":"a96e7282-4793-4de4-b2a0-deb12e53fb4c","Type":"ContainerStarted","Data":"d9c9b63c175e055bcaa2f807d9d9991d40650d68cb8d64bbdc8aeb5c9d21f4d5"} Jul 22 20:02:02.117720 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.117691 2575 generic.go:356] "Generic (PLEG): container finished" podID="205d0975-9993-4a3e-ae60-573dda5e4d73" containerID="41795c7ec3a3d9a47f9767344831fbf664e5ef5b518a763885cf1973ab4d6417" exitCode=0 Jul 22 20:02:02.117800 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.117772 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlwgk" event={"ID":"205d0975-9993-4a3e-ae60-573dda5e4d73","Type":"ContainerDied","Data":"41795c7ec3a3d9a47f9767344831fbf664e5ef5b518a763885cf1973ab4d6417"} Jul 22 20:02:02.119439 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.119409 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" event={"ID":"193d7f53-af94-46a2-ad8b-97786c97081b","Type":"ContainerStarted","Data":"ae45d83fdbedd4f175342b6b41a85329a4dd193733080d312d4500cd3b9123ac"} Jul 22 20:02:02.121345 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.121325 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" event={"ID":"89a76f54cfe80502fbc7b805a8de1eda","Type":"ContainerStarted","Data":"2f3ddd01b908125d9145cabf0ec9824150a7b6f79717f02721bdecf2d00a1470"} Jul 22 20:02:02.123189 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.123134 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" event={"ID":"a108f216-facb-45e9-b456-516323826a5b","Type":"ContainerStarted","Data":"5605619ece470f45c27526e83887fe1b8821b24cf7f659b006d8194facec4b2e"} Jul 22 20:02:02.123709 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.123660 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f99nc" podStartSLOduration=6.446386356 podStartE2EDuration="22.123643399s" totalImagesPullingTime="15.677257043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:02.122736799 +0000 UTC m=+22.635616510" watchObservedRunningTime="2026-07-22 20:02:02.123643399 +0000 UTC m=+22.636523106" Jul 22 20:02:02.124711 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.124692 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7jg4v" event={"ID":"90fd0be7-5de1-41c4-aeb0-59685a8907c1","Type":"ContainerStarted","Data":"e2f8ef840bc3b6e16c16f04529ee221424d08f89fdf066e5586fbb56dafbb2d1"} Jul 22 20:02:02.126289 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.126267 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b9gqf" event={"ID":"21f357d8-3cda-47c1-b91b-1bc4fcf45113","Type":"ContainerStarted","Data":"1967dd1cd170726e905842c41b593e5b34b22916f664da2cfe1bf6716d4fecc0"} Jul 22 20:02:02.138098 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.138043 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-196.ec2.internal" podStartSLOduration=22.138023006 podStartE2EDuration="22.138023006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:02.137318798 +0000 UTC m=+22.650198504" watchObservedRunningTime="2026-07-22 20:02:02.138023006 +0000 UTC m=+22.650902713" Jul 22 20:02:02.150076 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.150022 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-196.ec2.internal" podStartSLOduration=22.150006972 podStartE2EDuration="22.150006972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:02.149791288 +0000 UTC m=+22.662670995" watchObservedRunningTime="2026-07-22 20:02:02.150006972 +0000 UTC m=+22.662886677" Jul 22 20:02:02.199836 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.199777 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rk8bn" podStartSLOduration=2.52258241 podStartE2EDuration="22.199755133s" totalImagesPullingTime="19.677172723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:02.199381749 +0000 UTC m=+22.712261456" watchObservedRunningTime="2026-07-22 20:02:02.199755133 +0000 UTC m=+22.712634840" Jul 22 20:02:02.200009 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.199915 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5sdbv" podStartSLOduration=11.16740602 podStartE2EDuration="22.199908496s" totalImagesPullingTime="11.032502476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:02.182807703 +0000 UTC m=+22.695687411" watchObservedRunningTime="2026-07-22 20:02:02.199908496 +0000 UTC m=+22.712788202" Jul 22 20:02:02.219590 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.219543 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-b9gqf" podStartSLOduration=2.463157641 podStartE2EDuration="22.219525016s" totalImagesPullingTime="19.756367375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:02.21897217 +0000 UTC m=+22.731851877" watchObservedRunningTime="2026-07-22 20:02:02.219525016 +0000 UTC m=+22.732404723" Jul 22 20:02:02.234218 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.234176 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7jg4v" podStartSLOduration=2.54366408 podStartE2EDuration="22.234161516s" totalImagesPullingTime="19.690497436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:02.233820971 +0000 UTC m=+22.746700676" watchObservedRunningTime="2026-07-22 20:02:02.234161516 +0000 UTC m=+22.747041221" Jul 22 20:02:02.425996 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.425887 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7jg4v" Jul 22 20:02:02.871455 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.871436 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 20:02:02.928063 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.927946 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T20:02:02.871452253Z","UUID":"d0f15339-b67e-4d0d-adae-071d7d015f91","Handler":null,"Name":"","Endpoint":""} Jul 22 20:02:02.929209 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.929193 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 20:02:02.929292 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:02.929224 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 20:02:03.001820 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:03.001782 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:02:03.001820 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:03.001807 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:02:03.002040 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:03.001896 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:02:03.002040 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:03.001950 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:02:03.002110 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:03.002047 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:02:03.002142 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:03.002124 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:02:03.130310 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:03.130231 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" event={"ID":"193d7f53-af94-46a2-ad8b-97786c97081b","Type":"ContainerStarted","Data":"39a6f586721e8d1f9009079e3151bc698de6098c71bbdb2d78cab2e59c03925d"} Jul 22 20:02:03.131686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:03.131645 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r2lsd" event={"ID":"56ff2751-6545-46b9-860b-2d3d44c516e5","Type":"ContainerStarted","Data":"ab99b3232b5cfecebb30d9d8b8d4f38bdf3fd9b5abe06385cde64cb62f801a72"} Jul 22 20:02:03.149603 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:03.149564 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r2lsd" podStartSLOduration=14.334654357 podStartE2EDuration="23.149553075s" totalImagesPullingTime="8.814898718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:03.149432567 +0000 UTC m=+23.662312288" watchObservedRunningTime="2026-07-22 20:02:03.149553075 +0000 UTC m=+23.662432779" Jul 22 20:02:05.001632 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:05.001458 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:02:05.002110 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:05.001511 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:02:05.002110 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:05.001722 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:02:05.002110 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:05.001511 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:02:05.002110 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:05.001773 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:02:05.002110 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:05.001880 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:02:05.139536 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:05.139509 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:02:05.139947 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:05.139908 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" event={"ID":"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b","Type":"ContainerStarted","Data":"7d07c1b08622a039990daf11cdce047f37c943aeed4d1980fbd738be14df110b"} Jul 22 20:02:05.591769 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:05.591726 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7jg4v" Jul 22 20:02:05.592696 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:05.592670 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7jg4v" Jul 22 20:02:06.145041 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:06.145005 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" event={"ID":"193d7f53-af94-46a2-ad8b-97786c97081b","Type":"ContainerStarted","Data":"ab8c6cf1065f999a765ec4d2c1bcbe0f59f7a305f780e3e8e8d76488c2e0647f"} Jul 22 20:02:06.145831 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:06.145812 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7jg4v" Jul 22 20:02:06.166125 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:06.166076 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6k24" podStartSLOduration=10.631051711 podStartE2EDuration="26.166061048s" totalImagesPullingTime="15.535009337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:06.165314854 +0000 UTC m=+26.678194560" watchObservedRunningTime="2026-07-22 20:02:06.166061048 +0000 UTC m=+26.678940755" Jul 22 20:02:07.001374 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:07.001329 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:02:07.001374 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:07.001356 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:02:07.001646 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:07.001329 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:02:07.001646 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:07.001472 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:02:07.001646 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:07.001527 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:02:07.001646 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:07.001633 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:02:08.153121 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:08.152951 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:02:08.153597 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:08.153461 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" event={"ID":"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b","Type":"ContainerStarted","Data":"84992d90aeae21128d3d7a7d08c4252ef69ff61602ecc960fad7d40649c0d725"} Jul 22 20:02:08.153695 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:08.153668 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:02:08.153799 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:08.153733 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:02:08.153799 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:08.153780 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:02:08.153942 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:08.153911 2575 scope.go:117] "RemoveContainer" containerID="7d0e406d043d3ea9050baea421000895345c3aaf28ade29982dd85fca2265030" Jul 22 20:02:08.181596 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:08.181570 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:02:08.181861 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:08.181844 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:02:08.295019 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:08.294971 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:08.380366 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:08.380293 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:08.981402 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:08.981371 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:09.001449 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.001424 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:02:09.001604 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.001454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:02:09.001604 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.001424 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:02:09.001604 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:09.001529 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:02:09.001726 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:09.001615 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:02:09.001726 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:09.001668 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:02:09.047177 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.047098 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:09.148163 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.148131 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:09.159216 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.159194 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:02:09.159591 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.159541 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" event={"ID":"ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b","Type":"ContainerStarted","Data":"64987b89c05e62851407acd80bc274181c8dc2518e53d2ca35c543432bddc9cf"} Jul 22 20:02:09.161260 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.161233 2575 generic.go:356] "Generic (PLEG): container finished" podID="205d0975-9993-4a3e-ae60-573dda5e4d73" containerID="0d7409ff15213edf92238c5595727d89264220ced382364c1d4881565c68beb2" exitCode=0 Jul 22 20:02:09.161369 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.161280 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlwgk" event={"ID":"205d0975-9993-4a3e-ae60-573dda5e4d73","Type":"ContainerDied","Data":"0d7409ff15213edf92238c5595727d89264220ced382364c1d4881565c68beb2"} Jul 22 20:02:09.216596 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.216548 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" podStartSLOduration=9.000034219 podStartE2EDuration="29.216534236s" totalImagesPullingTime="20.216500017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:09.19981621 +0000 UTC m=+29.712695915" watchObservedRunningTime="2026-07-22 20:02:09.216534236 +0000 UTC m=+29.729413942" Jul 22 20:02:09.992016 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.991963 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j7h4z"] Jul 22 20:02:09.992189 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.992100 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:02:09.992247 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:09.992193 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:02:09.992537 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.992515 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6r6fq"] Jul 22 20:02:09.992635 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:09.992585 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:02:09.992673 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:09.992651 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:02:10.000531 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:10.000474 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:10.004611 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:10.004590 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29ngf"] Jul 22 20:02:10.004737 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:10.004683 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:02:10.004812 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:10.004790 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:02:10.063962 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:10.063927 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:10.998582 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:10.998370 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:11.168824 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:11.168791 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:11.269834 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:11.269712 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:12.001115 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:12.001075 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:02:12.001115 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:12.001108 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:02:12.001569 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:12.001075 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:02:12.001569 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:12.001206 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:02:12.001569 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:12.001319 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7h4z" podUID="67099d8a-f2c3-492f-a99f-6d81f030bc54" Jul 22 20:02:12.001569 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:12.001376 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6r6fq" podUID="2a4698b9-5706-4044-805e-be927c65786b" Jul 22 20:02:12.706493 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:12.706463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:02:12.706683 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:12.706524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:02:12.706683 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:12.706602 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:02:12.706683 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:12.706633 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 20:02:12.706683 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:12.706663 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs podName:d77c3889-b5fa-491e-b122-3b1c06ac4bd0 nodeName:}" failed. No retries permitted until 2026-07-22 20:02:44.706647055 +0000 UTC m=+65.219526740 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs") pod "network-metrics-daemon-29ngf" (UID: "d77c3889-b5fa-491e-b122-3b1c06ac4bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 20:02:12.706683 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:12.706678 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret podName:67099d8a-f2c3-492f-a99f-6d81f030bc54 nodeName:}" failed. No retries permitted until 2026-07-22 20:02:44.706671435 +0000 UTC m=+65.219551119 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret") pod "global-pull-secret-syncer-j7h4z" (UID: "67099d8a-f2c3-492f-a99f-6d81f030bc54") : object "kube-system"/"original-pull-secret" not registered Jul 22 20:02:12.807095 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:12.807061 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhld\" (UniqueName: \"kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld\") pod \"network-check-target-6r6fq\" (UID: \"2a4698b9-5706-4044-805e-be927c65786b\") " pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:02:12.807259 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:12.807241 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 20:02:12.807337 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:12.807269 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 20:02:12.807337 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:12.807283 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hvhld for pod openshift-network-diagnostics/network-check-target-6r6fq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:02:12.807429 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:12.807349 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld podName:2a4698b9-5706-4044-805e-be927c65786b nodeName:}" failed. No retries permitted until 2026-07-22 20:02:44.807329031 +0000 UTC m=+65.320208729 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvhld" (UniqueName: "kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld") pod "network-check-target-6r6fq" (UID: "2a4698b9-5706-4044-805e-be927c65786b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 20:02:13.008179 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.008080 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:13.099089 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.099057 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:13.171922 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.171892 2575 generic.go:356] "Generic (PLEG): container finished" podID="205d0975-9993-4a3e-ae60-573dda5e4d73" containerID="42a9e8f8e8c062241c749667867a66c7596d37412df9da73399464f8a417b6bd" exitCode=0 Jul 22 20:02:13.172104 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.171939 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlwgk" event={"ID":"205d0975-9993-4a3e-ae60-573dda5e4d73","Type":"ContainerDied","Data":"42a9e8f8e8c062241c749667867a66c7596d37412df9da73399464f8a417b6bd"} Jul 22 20:02:13.831934 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.831865 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-196.ec2.internal" event="NodeReady" Jul 22 20:02:13.832076 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.832017 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 20:02:13.875603 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.875576 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xlzdp"] Jul 22 20:02:13.886780 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.886735 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sw289"] Jul 22 20:02:13.886920 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.886872 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:13.889763 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.889738 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 20:02:13.889924 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.889746 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zbn42\"" Jul 22 20:02:13.890276 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.890256 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 20:02:13.896892 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.896872 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xlzdp"] Jul 22 20:02:13.897007 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.896902 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sw289"] Jul 22 20:02:13.897007 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.896995 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sw289" Jul 22 20:02:13.899266 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.899241 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 20:02:13.899366 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.899246 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 20:02:13.899366 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.899282 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9cx5s\"" Jul 22 20:02:13.899366 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.899308 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 20:02:13.986854 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:13.986821 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:14.001885 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.001859 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:02:14.002037 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.001859 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:02:14.002037 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.002006 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:02:14.004417 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.004376 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 20:02:14.004417 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.004415 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nrd5t\"" Jul 22 20:02:14.004619 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.004537 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 20:02:14.004673 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.004629 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 20:02:14.004718 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.004684 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 20:02:14.004792 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.004771 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l85mx\"" Jul 22 20:02:14.017011 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.016966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7scr\" (UniqueName: \"kubernetes.io/projected/2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116-kube-api-access-c7scr\") pod \"ingress-canary-sw289\" (UID: \"2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116\") " pod="openshift-ingress-canary/ingress-canary-sw289" Jul 22 20:02:14.017433 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.017022 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a01377ab-7b6f-4775-a9ef-260b8216c7a4-config-volume\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:14.017433 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.017058 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116-cert\") pod \"ingress-canary-sw289\" (UID: \"2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116\") " pod="openshift-ingress-canary/ingress-canary-sw289" Jul 22 20:02:14.017433 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.017078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6nht\" (UniqueName: \"kubernetes.io/projected/a01377ab-7b6f-4775-a9ef-260b8216c7a4-kube-api-access-j6nht\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:14.017433 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.017113 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:14.017433 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.017131 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a01377ab-7b6f-4775-a9ef-260b8216c7a4-tmp-dir\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:14.117525 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.117441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7scr\" (UniqueName: \"kubernetes.io/projected/2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116-kube-api-access-c7scr\") pod \"ingress-canary-sw289\" (UID: \"2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116\") " pod="openshift-ingress-canary/ingress-canary-sw289" Jul 22 20:02:14.117525 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.117481 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a01377ab-7b6f-4775-a9ef-260b8216c7a4-config-volume\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:14.117525 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.117511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116-cert\") pod \"ingress-canary-sw289\" (UID: \"2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116\") " pod="openshift-ingress-canary/ingress-canary-sw289" Jul 22 20:02:14.117525 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.117527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6nht\" (UniqueName: \"kubernetes.io/projected/a01377ab-7b6f-4775-a9ef-260b8216c7a4-kube-api-access-j6nht\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:14.117822 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.117592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:14.117822 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.117619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a01377ab-7b6f-4775-a9ef-260b8216c7a4-tmp-dir\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:14.117822 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:14.117698 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 20:02:14.117822 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:14.117769 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls podName:a01377ab-7b6f-4775-a9ef-260b8216c7a4 nodeName:}" failed. No retries permitted until 2026-07-22 20:02:14.617748525 +0000 UTC m=+35.130628213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls") pod "dns-default-xlzdp" (UID: "a01377ab-7b6f-4775-a9ef-260b8216c7a4") : secret "dns-default-metrics-tls" not found Jul 22 20:02:14.118045 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.118035 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a01377ab-7b6f-4775-a9ef-260b8216c7a4-tmp-dir\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:14.118182 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.118164 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a01377ab-7b6f-4775-a9ef-260b8216c7a4-config-volume\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:14.121750 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.121724 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116-cert\") pod \"ingress-canary-sw289\" (UID: \"2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116\") " pod="openshift-ingress-canary/ingress-canary-sw289" Jul 22 20:02:14.125957 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.125925 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7scr\" (UniqueName: \"kubernetes.io/projected/2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116-kube-api-access-c7scr\") pod \"ingress-canary-sw289\" (UID: \"2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116\") " pod="openshift-ingress-canary/ingress-canary-sw289" Jul 22 20:02:14.126075 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.126013 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6nht\" (UniqueName: \"kubernetes.io/projected/a01377ab-7b6f-4775-a9ef-260b8216c7a4-kube-api-access-j6nht\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:14.205115 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.205089 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sw289" Jul 22 20:02:14.397132 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.396869 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sw289"] Jul 22 20:02:14.401895 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:02:14.401860 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f07cfd6_24f9_4d03_a5f6_5c4bc1c83116.slice/crio-bd4cb79e6cc5af84add1c3cf6347087947cb5b33be3874f26fcec6a566110da1 WatchSource:0}: Error finding container bd4cb79e6cc5af84add1c3cf6347087947cb5b33be3874f26fcec6a566110da1: Status 404 returned error can't find the container with id bd4cb79e6cc5af84add1c3cf6347087947cb5b33be3874f26fcec6a566110da1 Jul 22 20:02:14.621182 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:14.621142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:14.621348 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:14.621278 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 20:02:14.621348 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:14.621340 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls podName:a01377ab-7b6f-4775-a9ef-260b8216c7a4 nodeName:}" failed. No retries permitted until 2026-07-22 20:02:15.621325408 +0000 UTC m=+36.134205092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls") pod "dns-default-xlzdp" (UID: "a01377ab-7b6f-4775-a9ef-260b8216c7a4") : secret "dns-default-metrics-tls" not found Jul 22 20:02:15.179549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:15.179510 2575 generic.go:356] "Generic (PLEG): container finished" podID="205d0975-9993-4a3e-ae60-573dda5e4d73" containerID="d0ef22ba0f277fe5b214004a88e43a9c049cc9108f917e1a0d5fc8c95dcc706c" exitCode=0 Jul 22 20:02:15.180288 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:15.179595 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlwgk" event={"ID":"205d0975-9993-4a3e-ae60-573dda5e4d73","Type":"ContainerDied","Data":"d0ef22ba0f277fe5b214004a88e43a9c049cc9108f917e1a0d5fc8c95dcc706c"} Jul 22 20:02:15.180722 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:15.180698 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sw289" event={"ID":"2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116","Type":"ContainerStarted","Data":"bd4cb79e6cc5af84add1c3cf6347087947cb5b33be3874f26fcec6a566110da1"} Jul 22 20:02:15.386905 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:15.386820 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:15.442449 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:15.442393 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:15.630540 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:15.630498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:15.630709 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:15.630645 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 20:02:15.630762 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:15.630711 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls podName:a01377ab-7b6f-4775-a9ef-260b8216c7a4 nodeName:}" failed. No retries permitted until 2026-07-22 20:02:17.630689668 +0000 UTC m=+38.143569353 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls") pod "dns-default-xlzdp" (UID: "a01377ab-7b6f-4775-a9ef-260b8216c7a4") : secret "dns-default-metrics-tls" not found Jul 22 20:02:17.187635 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:17.187516 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sw289" event={"ID":"2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116","Type":"ContainerStarted","Data":"08e96e047c24238e0134dedbe660d1db4629698440ce335220b3b417c7e66c02"} Jul 22 20:02:17.202815 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:17.202760 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sw289" podStartSLOduration=1.729935462 podStartE2EDuration="4.202741001s" totalImagesPullingTime="2.472805539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:17.201172011 +0000 UTC m=+37.714051718" watchObservedRunningTime="2026-07-22 20:02:17.202741001 +0000 UTC m=+37.715620708" Jul 22 20:02:17.646383 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:17.646280 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:17.646535 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:17.646433 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 20:02:17.646535 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:17.646490 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls podName:a01377ab-7b6f-4775-a9ef-260b8216c7a4 nodeName:}" failed. No retries permitted until 2026-07-22 20:02:21.646477808 +0000 UTC m=+42.159357493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls") pod "dns-default-xlzdp" (UID: "a01377ab-7b6f-4775-a9ef-260b8216c7a4") : secret "dns-default-metrics-tls" not found Jul 22 20:02:19.146023 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:19.145968 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:19.604259 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:19.604227 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:21.252091 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:21.252057 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:21.307138 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:21.307115 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:21.679678 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:21.679638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:21.679851 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:21.679801 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 20:02:21.679899 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:21.679881 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls podName:a01377ab-7b6f-4775-a9ef-260b8216c7a4 nodeName:}" failed. No retries permitted until 2026-07-22 20:02:29.679863596 +0000 UTC m=+50.192743309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls") pod "dns-default-xlzdp" (UID: "a01377ab-7b6f-4775-a9ef-260b8216c7a4") : secret "dns-default-metrics-tls" not found Jul 22 20:02:22.203464 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:22.203432 2575 generic.go:356] "Generic (PLEG): container finished" podID="205d0975-9993-4a3e-ae60-573dda5e4d73" containerID="d10ab60fbc7457fb3ce5d76af50a7489002404fc8f278d7ed475fc41f334c1f7" exitCode=0 Jul 22 20:02:22.203638 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:22.203480 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlwgk" event={"ID":"205d0975-9993-4a3e-ae60-573dda5e4d73","Type":"ContainerDied","Data":"d10ab60fbc7457fb3ce5d76af50a7489002404fc8f278d7ed475fc41f334c1f7"} Jul 22 20:02:23.208340 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:23.208308 2575 generic.go:356] "Generic (PLEG): container finished" podID="205d0975-9993-4a3e-ae60-573dda5e4d73" containerID="f691c947c554e00ed7f19fc6c04621de37bf305a7c8a8cca3108c5b704093a95" exitCode=0 Jul 22 20:02:23.208677 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:23.208358 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlwgk" event={"ID":"205d0975-9993-4a3e-ae60-573dda5e4d73","Type":"ContainerDied","Data":"f691c947c554e00ed7f19fc6c04621de37bf305a7c8a8cca3108c5b704093a95"} Jul 22 20:02:24.213593 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:24.213558 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlwgk" event={"ID":"205d0975-9993-4a3e-ae60-573dda5e4d73","Type":"ContainerStarted","Data":"60eab3ffe893f32438feb353c4f69d70465e17117f7ca78710b9e5c8da5c8769"} Jul 22 20:02:24.235052 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:24.235003 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jlwgk" podStartSLOduration=8.43096896 podStartE2EDuration="44.234972282s" totalImagesPullingTime="35.804003322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:24.233607076 +0000 UTC m=+44.746486782" watchObservedRunningTime="2026-07-22 20:02:24.234972282 +0000 UTC m=+44.747851987" Jul 22 20:02:24.357035 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:24.356993 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:24.419657 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:24.419630 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:24.532483 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:24.532386 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:26.387484 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:26.387447 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:26.487888 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:26.487856 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:29.736482 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:29.736440 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:29.736881 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:29.736593 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 20:02:29.736881 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:29.736659 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls podName:a01377ab-7b6f-4775-a9ef-260b8216c7a4 nodeName:}" failed. No retries permitted until 2026-07-22 20:02:45.736643627 +0000 UTC m=+66.249523311 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls") pod "dns-default-xlzdp" (UID: "a01377ab-7b6f-4775-a9ef-260b8216c7a4") : secret "dns-default-metrics-tls" not found Jul 22 20:02:32.342141 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:32.342108 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:32.397590 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:32.397566 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:33.124312 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:33.124273 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:33.170955 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:33.170925 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:40.178670 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:40.178643 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zsbtd" Jul 22 20:02:40.341415 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:40.341377 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:40.405951 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:40.405917 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:44.741236 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:44.741193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:02:44.741627 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:44.741253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:02:44.743785 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:44.743764 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 20:02:44.743873 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:44.743857 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 20:02:44.751684 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:44.751663 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 20:02:44.751751 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:44.751740 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs podName:d77c3889-b5fa-491e-b122-3b1c06ac4bd0 nodeName:}" failed. No retries permitted until 2026-07-22 20:03:48.751720147 +0000 UTC m=+129.264599833 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs") pod "network-metrics-daemon-29ngf" (UID: "d77c3889-b5fa-491e-b122-3b1c06ac4bd0") : secret "metrics-daemon-secret" not found Jul 22 20:02:44.754970 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:44.754939 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67099d8a-f2c3-492f-a99f-6d81f030bc54-original-pull-secret\") pod \"global-pull-secret-syncer-j7h4z\" (UID: \"67099d8a-f2c3-492f-a99f-6d81f030bc54\") " pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:02:44.842196 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:44.842158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhld\" (UniqueName: \"kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld\") pod \"network-check-target-6r6fq\" (UID: \"2a4698b9-5706-4044-805e-be927c65786b\") " pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:02:44.844785 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:44.844766 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 20:02:44.854756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:44.854734 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 20:02:44.865388 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:44.865362 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvhld\" (UniqueName: \"kubernetes.io/projected/2a4698b9-5706-4044-805e-be927c65786b-kube-api-access-hvhld\") pod \"network-check-target-6r6fq\" (UID: \"2a4698b9-5706-4044-805e-be927c65786b\") " pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:02:44.911756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:44.911717 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7h4z" Jul 22 20:02:44.925085 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:44.925054 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nrd5t\"" Jul 22 20:02:44.933456 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:44.933432 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:02:45.057880 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.057722 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j7h4z"] Jul 22 20:02:45.061431 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:02:45.061408 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67099d8a_f2c3_492f_a99f_6d81f030bc54.slice/crio-52a9254baeabae74130efc7455f7502664184303be2cb8aa301bdbf953728172 WatchSource:0}: Error finding container 52a9254baeabae74130efc7455f7502664184303be2cb8aa301bdbf953728172: Status 404 returned error can't find the container with id 52a9254baeabae74130efc7455f7502664184303be2cb8aa301bdbf953728172 Jul 22 20:02:45.074000 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.073963 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6r6fq"] Jul 22 20:02:45.077049 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:02:45.077020 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4698b9_5706_4044_805e_be927c65786b.slice/crio-08a0ebf37350b5e2d774d39af38e376b874fb9a11b7b14b1c5a030ef0a975c85 WatchSource:0}: Error finding container 08a0ebf37350b5e2d774d39af38e376b874fb9a11b7b14b1c5a030ef0a975c85: Status 404 returned error can't find the container with id 08a0ebf37350b5e2d774d39af38e376b874fb9a11b7b14b1c5a030ef0a975c85 Jul 22 20:02:45.261017 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.260902 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6r6fq" event={"ID":"2a4698b9-5706-4044-805e-be927c65786b","Type":"ContainerStarted","Data":"08a0ebf37350b5e2d774d39af38e376b874fb9a11b7b14b1c5a030ef0a975c85"} Jul 22 20:02:45.262025 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.262002 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j7h4z" event={"ID":"67099d8a-f2c3-492f-a99f-6d81f030bc54","Type":"ContainerStarted","Data":"52a9254baeabae74130efc7455f7502664184303be2cb8aa301bdbf953728172"} Jul 22 20:02:45.483883 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.483850 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5"] Jul 22 20:02:45.485897 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.485870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5" Jul 22 20:02:45.488284 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.488259 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx"] Jul 22 20:02:45.489090 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.489058 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 20:02:45.490004 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.489960 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 22 20:02:45.490004 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.489987 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 20:02:45.490194 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.490003 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-nkk8m\"" Jul 22 20:02:45.490194 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.490037 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 20:02:45.490755 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.490735 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.493486 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.493450 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 22 20:02:45.493631 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.493610 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 22 20:02:45.493801 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.493780 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 22 20:02:45.494235 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.494188 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 22 20:02:45.496860 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.496091 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5"] Jul 22 20:02:45.507249 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.507227 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx"] Jul 22 20:02:45.648462 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.648427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktgh7\" (UniqueName: \"kubernetes.io/projected/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-kube-api-access-ktgh7\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.648649 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.648499 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-hub\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.648649 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.648524 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.648649 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.648552 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.648649 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.648596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2997b\" (UniqueName: \"kubernetes.io/projected/1f68197b-fbdb-4478-8059-174acd3f4a84-kube-api-access-2997b\") pod \"managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5\" (UID: \"1f68197b-fbdb-4478-8059-174acd3f4a84\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5" Jul 22 20:02:45.648649 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.648647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f68197b-fbdb-4478-8059-174acd3f4a84-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5\" (UID: \"1f68197b-fbdb-4478-8059-174acd3f4a84\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5" Jul 22 20:02:45.648914 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.648673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-ca\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.648914 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.648695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.750026 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.749949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2997b\" (UniqueName: \"kubernetes.io/projected/1f68197b-fbdb-4478-8059-174acd3f4a84-kube-api-access-2997b\") pod \"managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5\" (UID: \"1f68197b-fbdb-4478-8059-174acd3f4a84\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5" Jul 22 20:02:45.750486 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.750130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:02:45.750486 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.750192 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f68197b-fbdb-4478-8059-174acd3f4a84-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5\" (UID: \"1f68197b-fbdb-4478-8059-174acd3f4a84\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5" Jul 22 20:02:45.750486 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.750214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-ca\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.750486 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.750234 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.750486 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.750267 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktgh7\" (UniqueName: \"kubernetes.io/projected/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-kube-api-access-ktgh7\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.750486 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:45.750366 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 20:02:45.750486 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:02:45.750465 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls podName:a01377ab-7b6f-4775-a9ef-260b8216c7a4 nodeName:}" failed. No retries permitted until 2026-07-22 20:03:17.75044262 +0000 UTC m=+98.263322306 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls") pod "dns-default-xlzdp" (UID: "a01377ab-7b6f-4775-a9ef-260b8216c7a4") : secret "dns-default-metrics-tls" not found Jul 22 20:02:45.750847 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.750543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-hub\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.750847 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.750571 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.750847 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.750605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.751521 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.751462 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.753619 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.753578 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.753726 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.753616 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.753854 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.753811 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f68197b-fbdb-4478-8059-174acd3f4a84-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5\" (UID: \"1f68197b-fbdb-4478-8059-174acd3f4a84\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5" Jul 22 20:02:45.754234 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.754209 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-ca\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.754320 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.754240 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-hub\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.758928 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.758904 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2997b\" (UniqueName: \"kubernetes.io/projected/1f68197b-fbdb-4478-8059-174acd3f4a84-kube-api-access-2997b\") pod \"managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5\" (UID: \"1f68197b-fbdb-4478-8059-174acd3f4a84\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5" Jul 22 20:02:45.759038 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.758903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktgh7\" (UniqueName: \"kubernetes.io/projected/1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2-kube-api-access-ktgh7\") pod \"cluster-proxy-proxy-agent-c874bfb96-8cqpx\" (UID: \"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.809651 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.809577 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5" Jul 22 20:02:45.815013 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.814967 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:02:45.971143 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.971081 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5"] Jul 22 20:02:45.974872 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:02:45.974841 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f68197b_fbdb_4478_8059_174acd3f4a84.slice/crio-d7b433ef81adf68bd33087c97add48a491e7a45f2f2955d89b7eb0dc86bb9013 WatchSource:0}: Error finding container d7b433ef81adf68bd33087c97add48a491e7a45f2f2955d89b7eb0dc86bb9013: Status 404 returned error can't find the container with id d7b433ef81adf68bd33087c97add48a491e7a45f2f2955d89b7eb0dc86bb9013 Jul 22 20:02:45.988342 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:45.988315 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx"] Jul 22 20:02:45.992012 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:02:45.991963 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d081a90_1e14_4bcd_92e7_0ce9dbbf14c2.slice/crio-7765d27da12e9eb27e88115e678cd08b0f0010c65ded9cb3ec9199bdcb4b1f64 WatchSource:0}: Error finding container 7765d27da12e9eb27e88115e678cd08b0f0010c65ded9cb3ec9199bdcb4b1f64: Status 404 returned error can't find the container with id 7765d27da12e9eb27e88115e678cd08b0f0010c65ded9cb3ec9199bdcb4b1f64 Jul 22 20:02:46.267055 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:46.267008 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" event={"ID":"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2","Type":"ContainerStarted","Data":"7765d27da12e9eb27e88115e678cd08b0f0010c65ded9cb3ec9199bdcb4b1f64"} Jul 22 20:02:46.268466 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:46.268437 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5" event={"ID":"1f68197b-fbdb-4478-8059-174acd3f4a84","Type":"ContainerStarted","Data":"d7b433ef81adf68bd33087c97add48a491e7a45f2f2955d89b7eb0dc86bb9013"} Jul 22 20:02:48.070378 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:48.070311 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:48.173524 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:48.173406 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:48.223834 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:48.223782 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:02:53.291196 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:53.291136 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6r6fq" event={"ID":"2a4698b9-5706-4044-805e-be927c65786b","Type":"ContainerStarted","Data":"d73689485b9f7452dd25dfff2e5aa307d132aafc02db6696ad931f180c6ca36c"} Jul 22 20:02:53.291681 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:53.291355 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:02:53.293212 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:53.293190 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j7h4z" event={"ID":"67099d8a-f2c3-492f-a99f-6d81f030bc54","Type":"ContainerStarted","Data":"15d40c2e13a45ed73631ac2026ffe1eadef60d8fa817eab10594280db50eb432"} Jul 22 20:02:53.295164 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:53.295141 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" event={"ID":"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2","Type":"ContainerStarted","Data":"7b65d9f411aff6d7fbed87ea55e5a7515ce11edbef08c8b0724ec89e10cfbb92"} Jul 22 20:02:53.295273 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:53.295173 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" event={"ID":"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2","Type":"ContainerStarted","Data":"03fc745dbb5b6f638c7c1350026c84511eb51a0ad58aee23ab7b0720d20e111f"} Jul 22 20:02:53.295273 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:53.295187 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" event={"ID":"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2","Type":"ContainerStarted","Data":"ed5d4f19ff5d24c533c75fd67805f7943c760b6bf40ef08703f2e704e93c8bbb"} Jul 22 20:02:53.296442 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:53.296414 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5" event={"ID":"1f68197b-fbdb-4478-8059-174acd3f4a84","Type":"ContainerStarted","Data":"bc2f3aa0e7ff69969231fb2d8abb6c56064821ee4cfc5b30c21459a33ce831bc"} Jul 22 20:02:53.306393 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:53.306352 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6r6fq" podStartSLOduration=66.011042775 podStartE2EDuration="1m13.306339616s" totalImagesPullingTime="7.295296841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:53.305868821 +0000 UTC m=+73.818748527" watchObservedRunningTime="2026-07-22 20:02:53.306339616 +0000 UTC m=+73.819219323" Jul 22 20:02:53.319752 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:53.319710 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j7h4z" podStartSLOduration=66.00847006 podStartE2EDuration="1m13.319694904s" totalImagesPullingTime="7.311224844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:53.318537193 +0000 UTC m=+73.831416899" watchObservedRunningTime="2026-07-22 20:02:53.319694904 +0000 UTC m=+73.832574604" Jul 22 20:02:53.333247 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:53.333193 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5" podStartSLOduration=1.923122818 podStartE2EDuration="8.333175317s" totalImagesPullingTime="6.410052499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:53.332921015 +0000 UTC m=+73.845800722" watchObservedRunningTime="2026-07-22 20:02:53.333175317 +0000 UTC m=+73.846055020" Jul 22 20:02:53.350298 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:02:53.350255 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" podStartSLOduration=1.951147993 podStartE2EDuration="8.35023942s" totalImagesPullingTime="6.399091427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:02:53.349704999 +0000 UTC m=+73.862584706" watchObservedRunningTime="2026-07-22 20:02:53.35023942 +0000 UTC m=+73.863119126" Jul 22 20:03:17.774786 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:03:17.774736 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:03:17.775308 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:03:17.774860 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 20:03:17.775308 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:03:17.774923 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls podName:a01377ab-7b6f-4775-a9ef-260b8216c7a4 nodeName:}" failed. No retries permitted until 2026-07-22 20:04:21.774907913 +0000 UTC m=+162.287787597 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls") pod "dns-default-xlzdp" (UID: "a01377ab-7b6f-4775-a9ef-260b8216c7a4") : secret "dns-default-metrics-tls" not found Jul 22 20:03:24.302589 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:03:24.302558 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6r6fq" Jul 22 20:03:48.782815 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:03:48.782769 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:03:48.783242 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:03:48.782904 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 20:03:48.783242 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:03:48.782964 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs podName:d77c3889-b5fa-491e-b122-3b1c06ac4bd0 nodeName:}" failed. No retries permitted until 2026-07-22 20:05:50.782947226 +0000 UTC m=+251.295826911 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs") pod "network-metrics-daemon-29ngf" (UID: "d77c3889-b5fa-491e-b122-3b1c06ac4bd0") : secret "metrics-daemon-secret" not found Jul 22 20:04:16.897228 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:04:16.897181 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xlzdp" podUID="a01377ab-7b6f-4775-a9ef-260b8216c7a4" Jul 22 20:04:17.017511 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:04:17.017472 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:04:17.540732 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:04:17.540706 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xlzdp" Jul 22 20:04:21.812785 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:04:21.812739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:04:21.813257 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:04:21.812874 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 20:04:21.813257 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:04:21.812950 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls podName:a01377ab-7b6f-4775-a9ef-260b8216c7a4 nodeName:}" failed. No retries permitted until 2026-07-22 20:06:23.81292801 +0000 UTC m=+284.325807712 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls") pod "dns-default-xlzdp" (UID: "a01377ab-7b6f-4775-a9ef-260b8216c7a4") : secret "dns-default-metrics-tls" not found Jul 22 20:04:28.001276 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:04:28.001235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:04:52.645170 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:04:52.645138 2575 generic.go:356] "Generic (PLEG): container finished" podID="1f68197b-fbdb-4478-8059-174acd3f4a84" containerID="bc2f3aa0e7ff69969231fb2d8abb6c56064821ee4cfc5b30c21459a33ce831bc" exitCode=255 Jul 22 20:04:52.645579 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:04:52.645214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5" event={"ID":"1f68197b-fbdb-4478-8059-174acd3f4a84","Type":"ContainerDied","Data":"bc2f3aa0e7ff69969231fb2d8abb6c56064821ee4cfc5b30c21459a33ce831bc"} Jul 22 20:04:52.645579 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:04:52.645517 2575 scope.go:117] "RemoveContainer" containerID="bc2f3aa0e7ff69969231fb2d8abb6c56064821ee4cfc5b30c21459a33ce831bc" Jul 22 20:04:53.649887 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:04:53.649851 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bc79bf48d-fc9k5" event={"ID":"1f68197b-fbdb-4478-8059-174acd3f4a84","Type":"ContainerStarted","Data":"3d438a96165af850dccf80b77842035d835633838a9d5dcbaf13ba00659c8860"} Jul 22 20:05:50.830052 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:50.829993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:05:50.830548 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:50.830151 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 20:05:50.830548 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:50.830218 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs podName:d77c3889-b5fa-491e-b122-3b1c06ac4bd0 nodeName:}" failed. No retries permitted until 2026-07-22 20:07:52.830202043 +0000 UTC m=+373.343081727 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs") pod "network-metrics-daemon-29ngf" (UID: "d77c3889-b5fa-491e-b122-3b1c06ac4bd0") : secret "metrics-daemon-secret" not found Jul 22 20:05:52.711867 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:52.711834 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:05:52.798413 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:52.798373 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:05:55.879416 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:55.879382 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:05:55.954607 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:55.954574 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:05:56.325489 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.325451 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-btl22"] Jul 22 20:05:56.327375 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.327359 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:56.330039 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.330007 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 20:05:56.330178 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.330030 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 20:05:56.330178 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.330009 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 20:05:56.330178 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.330122 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 20:05:56.330902 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.330889 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tfrxf\"" Jul 22 20:05:56.341735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.341709 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-btl22"] Jul 22 20:05:56.364688 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.364660 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:05:56.470672 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.470627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:56.470672 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.470683 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/677263c5-5fe3-40de-9004-02bb5bebf54b-crio-socket\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:56.470819 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.470717 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/677263c5-5fe3-40de-9004-02bb5bebf54b-data-volume\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:56.470819 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.470732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhpps\" (UniqueName: \"kubernetes.io/projected/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-api-access-xhpps\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:56.470819 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.470750 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:56.571680 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.571645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:56.571858 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.571685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/677263c5-5fe3-40de-9004-02bb5bebf54b-crio-socket\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:56.571858 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.571733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/677263c5-5fe3-40de-9004-02bb5bebf54b-data-volume\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:56.571858 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.571754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhpps\" (UniqueName: \"kubernetes.io/projected/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-api-access-xhpps\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:56.571858 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.571787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:56.571858 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:56.571801 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 20:05:56.572112 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.571853 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/677263c5-5fe3-40de-9004-02bb5bebf54b-crio-socket\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:56.572112 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:56.571866 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls podName:677263c5-5fe3-40de-9004-02bb5bebf54b nodeName:}" failed. No retries permitted until 2026-07-22 20:05:57.071847796 +0000 UTC m=+257.584727484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-btl22" (UID: "677263c5-5fe3-40de-9004-02bb5bebf54b") : secret "insights-runtime-extractor-tls" not found Jul 22 20:05:56.572112 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:56.571895 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 20:05:56.572112 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:56.571953 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm podName:677263c5-5fe3-40de-9004-02bb5bebf54b nodeName:}" failed. No retries permitted until 2026-07-22 20:05:57.071941082 +0000 UTC m=+257.584820772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-btl22" (UID: "677263c5-5fe3-40de-9004-02bb5bebf54b") : configmap "kube-rbac-proxy" not found Jul 22 20:05:56.572252 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.572144 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/677263c5-5fe3-40de-9004-02bb5bebf54b-data-volume\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:56.579745 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:56.579700 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhpps\" (UniqueName: \"kubernetes.io/projected/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-api-access-xhpps\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:57.077140 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:57.077103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:57.077757 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:57.077733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:57.078064 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:57.078047 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 20:05:57.078227 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:57.078217 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls podName:677263c5-5fe3-40de-9004-02bb5bebf54b nodeName:}" failed. No retries permitted until 2026-07-22 20:05:58.078197039 +0000 UTC m=+258.591076739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-btl22" (UID: "677263c5-5fe3-40de-9004-02bb5bebf54b") : secret "insights-runtime-extractor-tls" not found Jul 22 20:05:57.078828 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:57.078806 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 20:05:57.078997 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:57.078968 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm podName:677263c5-5fe3-40de-9004-02bb5bebf54b nodeName:}" failed. No retries permitted until 2026-07-22 20:05:58.07894948 +0000 UTC m=+258.591829167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-btl22" (UID: "677263c5-5fe3-40de-9004-02bb5bebf54b") : configmap "kube-rbac-proxy" not found Jul 22 20:05:58.087030 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:58.086972 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:58.087415 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:58.087075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:05:58.087415 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:58.087129 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 20:05:58.087415 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:58.087149 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 20:05:58.087415 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:58.087211 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls podName:677263c5-5fe3-40de-9004-02bb5bebf54b nodeName:}" failed. No retries permitted until 2026-07-22 20:06:00.087190106 +0000 UTC m=+260.600069794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-btl22" (UID: "677263c5-5fe3-40de-9004-02bb5bebf54b") : secret "insights-runtime-extractor-tls" not found Jul 22 20:05:58.087415 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:05:58.087228 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm podName:677263c5-5fe3-40de-9004-02bb5bebf54b nodeName:}" failed. No retries permitted until 2026-07-22 20:06:00.087220699 +0000 UTC m=+260.600100382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-btl22" (UID: "677263c5-5fe3-40de-9004-02bb5bebf54b") : configmap "kube-rbac-proxy" not found Jul 22 20:05:58.104120 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:58.104099 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f99nc_88aa1673-501e-43a1-8e35-20b2a70950d0/dns-node-resolver/0.log" Jul 22 20:05:58.152781 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:58.152749 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:05:58.409878 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:58.409795 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:05:58.511342 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:58.511308 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:05:58.761582 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:58.761548 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:05:58.818875 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:58.818847 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:05:59.061948 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:59.061872 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:05:59.102053 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:59.102031 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5sdbv_a96e7282-4793-4de4-b2a0-deb12e53fb4c/node-ca/0.log" Jul 22 20:05:59.161714 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:05:59.161685 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:00.103768 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:00.103717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:06:00.104311 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:00.103823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:06:00.104311 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:06:00.103844 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 20:06:00.104311 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:06:00.103904 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm podName:677263c5-5fe3-40de-9004-02bb5bebf54b nodeName:}" failed. No retries permitted until 2026-07-22 20:06:04.103887517 +0000 UTC m=+264.616767201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-btl22" (UID: "677263c5-5fe3-40de-9004-02bb5bebf54b") : configmap "kube-rbac-proxy" not found Jul 22 20:06:00.104311 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:06:00.103945 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 20:06:00.104311 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:06:00.104027 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls podName:677263c5-5fe3-40de-9004-02bb5bebf54b nodeName:}" failed. No retries permitted until 2026-07-22 20:06:04.104008949 +0000 UTC m=+264.616888642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-btl22" (UID: "677263c5-5fe3-40de-9004-02bb5bebf54b") : secret "insights-runtime-extractor-tls" not found Jul 22 20:06:00.302584 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:00.302555 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sw289_2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116/serve-healthcheck-canary/0.log" Jul 22 20:06:00.723036 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:00.723001 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:00.804768 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:00.804741 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:01.407302 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:01.407220 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:01.943158 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:01.943124 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:04.134475 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:04.134438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:06:04.134859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:04.134508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:06:04.134859 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:06:04.134591 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 20:06:04.134859 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:06:04.134672 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls podName:677263c5-5fe3-40de-9004-02bb5bebf54b nodeName:}" failed. No retries permitted until 2026-07-22 20:06:12.134657238 +0000 UTC m=+272.647536922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-btl22" (UID: "677263c5-5fe3-40de-9004-02bb5bebf54b") : secret "insights-runtime-extractor-tls" not found Jul 22 20:06:04.134859 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:06:04.134594 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 20:06:04.134859 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:06:04.134748 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm podName:677263c5-5fe3-40de-9004-02bb5bebf54b nodeName:}" failed. No retries permitted until 2026-07-22 20:06:12.134734822 +0000 UTC m=+272.647614517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-btl22" (UID: "677263c5-5fe3-40de-9004-02bb5bebf54b") : configmap "kube-rbac-proxy" not found Jul 22 20:06:04.546495 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:04.546465 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:04.715086 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:04.715049 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:04.897999 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:04.897899 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:04.999105 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:04.999071 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:05.099552 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:05.099518 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:05.200818 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:05.200793 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:05.417652 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:05.417622 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:06.403214 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:06.403186 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:06.797249 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:06.797222 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:07.940106 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:07.940080 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:08.040799 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:08.040762 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:08.174716 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:08.174688 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:08.589705 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:08.589671 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:08.692694 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:08.692658 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:08.978370 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:08.978338 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:09.390204 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:09.390123 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:09.612949 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:09.612917 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:09.797790 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:09.797757 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:09.984583 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:09.984554 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:10.386797 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:10.386753 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:10.997574 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:10.997535 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:11.107465 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:11.107438 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:11.207451 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:11.207419 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:11.584862 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:11.584830 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:11.988737 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:11.988705 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:12.192639 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:12.192587 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:06:12.193048 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:12.192661 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:06:12.193048 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:06:12.192756 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 20:06:12.193048 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:06:12.192825 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls podName:677263c5-5fe3-40de-9004-02bb5bebf54b nodeName:}" failed. No retries permitted until 2026-07-22 20:06:28.192807936 +0000 UTC m=+288.705687637 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-btl22" (UID: "677263c5-5fe3-40de-9004-02bb5bebf54b") : secret "insights-runtime-extractor-tls" not found Jul 22 20:06:12.193183 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:12.193165 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/677263c5-5fe3-40de-9004-02bb5bebf54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:06:12.442130 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:12.442102 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:12.513151 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:12.513122 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:12.780616 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:12.780535 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:12.989648 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:12.989617 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:13.047922 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:13.047848 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:13.388525 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:13.388442 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:13.581529 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:13.581491 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:14.180396 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:14.180357 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:14.580907 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:14.580872 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:15.604119 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:15.604090 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:16.731242 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:16.731212 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:16.864734 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:16.864704 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:16.962352 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:16.962320 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:17.046504 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:17.046427 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:18.151474 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:18.151440 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:18.252279 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:18.252249 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:18.560509 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:18.560479 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:18.913292 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:18.913206 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:19.013454 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.013424 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:19.216708 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.216667 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:19.318883 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.318854 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:19.624765 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.624696 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b478545d6-q76ss"] Jul 22 20:06:19.626596 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.626582 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.629237 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.629207 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gzjk5\"" Jul 22 20:06:19.629388 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.629254 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 20:06:19.629388 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.629254 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 20:06:19.629819 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.629804 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 20:06:19.635285 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.635255 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 20:06:19.645763 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.645740 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b478545d6-q76ss"] Jul 22 20:06:19.650114 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.650092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528a7576-71ce-438a-bfaa-e4e22be46b4d-trusted-ca\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.650193 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.650121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nfmh\" (UniqueName: \"kubernetes.io/projected/528a7576-71ce-438a-bfaa-e4e22be46b4d-kube-api-access-8nfmh\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.650193 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.650154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528a7576-71ce-438a-bfaa-e4e22be46b4d-registry-tls\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.650263 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.650233 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528a7576-71ce-438a-bfaa-e4e22be46b4d-installation-pull-secrets\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.650263 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.650259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528a7576-71ce-438a-bfaa-e4e22be46b4d-ca-trust-extracted\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.650320 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.650281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528a7576-71ce-438a-bfaa-e4e22be46b4d-image-registry-private-configuration\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.650320 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.650301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528a7576-71ce-438a-bfaa-e4e22be46b4d-registry-certificates\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.650382 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.650347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528a7576-71ce-438a-bfaa-e4e22be46b4d-bound-sa-token\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.665257 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.665237 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:19.751251 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.751230 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528a7576-71ce-438a-bfaa-e4e22be46b4d-trusted-ca\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.751337 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.751257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nfmh\" (UniqueName: \"kubernetes.io/projected/528a7576-71ce-438a-bfaa-e4e22be46b4d-kube-api-access-8nfmh\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.751337 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.751290 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528a7576-71ce-438a-bfaa-e4e22be46b4d-registry-tls\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.751423 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.751371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528a7576-71ce-438a-bfaa-e4e22be46b4d-installation-pull-secrets\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.751423 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.751399 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528a7576-71ce-438a-bfaa-e4e22be46b4d-ca-trust-extracted\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.751504 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.751426 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528a7576-71ce-438a-bfaa-e4e22be46b4d-image-registry-private-configuration\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.751504 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.751443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528a7576-71ce-438a-bfaa-e4e22be46b4d-registry-certificates\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.751504 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.751473 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528a7576-71ce-438a-bfaa-e4e22be46b4d-bound-sa-token\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.751865 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.751841 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528a7576-71ce-438a-bfaa-e4e22be46b4d-ca-trust-extracted\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.752268 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.752247 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528a7576-71ce-438a-bfaa-e4e22be46b4d-registry-certificates\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.752482 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.752458 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528a7576-71ce-438a-bfaa-e4e22be46b4d-trusted-ca\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.753880 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.753859 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528a7576-71ce-438a-bfaa-e4e22be46b4d-installation-pull-secrets\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.753964 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.753884 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528a7576-71ce-438a-bfaa-e4e22be46b4d-image-registry-private-configuration\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.753964 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.753871 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528a7576-71ce-438a-bfaa-e4e22be46b4d-registry-tls\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.761768 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.761744 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nfmh\" (UniqueName: \"kubernetes.io/projected/528a7576-71ce-438a-bfaa-e4e22be46b4d-kube-api-access-8nfmh\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.773617 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.773594 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528a7576-71ce-438a-bfaa-e4e22be46b4d-bound-sa-token\") pod \"image-registry-7b478545d6-q76ss\" (UID: \"528a7576-71ce-438a-bfaa-e4e22be46b4d\") " pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:19.776384 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.776365 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:19.936353 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:19.936277 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:20.059737 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:20.059710 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b478545d6-q76ss"] Jul 22 20:06:20.063586 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:06:20.063560 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528a7576_71ce_438a_bfaa_e4e22be46b4d.slice/crio-ae29c08fd99538fcf44dfbf990b4a9cefebd97aaad95612f6a58c5430dc650ad WatchSource:0}: Error finding container ae29c08fd99538fcf44dfbf990b4a9cefebd97aaad95612f6a58c5430dc650ad: Status 404 returned error can't find the container with id ae29c08fd99538fcf44dfbf990b4a9cefebd97aaad95612f6a58c5430dc650ad Jul 22 20:06:20.541650 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:06:20.541610 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xlzdp" podUID="a01377ab-7b6f-4775-a9ef-260b8216c7a4" Jul 22 20:06:20.892949 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:20.892865 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xlzdp" Jul 22 20:06:20.892949 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:20.892900 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b478545d6-q76ss" event={"ID":"528a7576-71ce-438a-bfaa-e4e22be46b4d","Type":"ContainerStarted","Data":"4d8c3158e4660ba4c9bdfa0abd5ceaa575570b764b6502606d06244c9d2ee785"} Jul 22 20:06:20.893164 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:20.892956 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b478545d6-q76ss" event={"ID":"528a7576-71ce-438a-bfaa-e4e22be46b4d","Type":"ContainerStarted","Data":"ae29c08fd99538fcf44dfbf990b4a9cefebd97aaad95612f6a58c5430dc650ad"} Jul 22 20:06:20.893164 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:20.893082 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:20.912320 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:20.912271 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b478545d6-q76ss" podStartSLOduration=1.912251592 podStartE2EDuration="1.912251592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:06:20.911729979 +0000 UTC m=+281.424609685" watchObservedRunningTime="2026-07-22 20:06:20.912251592 +0000 UTC m=+281.425131299" Jul 22 20:06:21.004952 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:21.004918 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:21.102856 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:21.102811 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:21.163201 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:21.163129 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:23.884952 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:23.884913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:06:23.887335 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:23.887312 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a01377ab-7b6f-4775-a9ef-260b8216c7a4-metrics-tls\") pod \"dns-default-xlzdp\" (UID: \"a01377ab-7b6f-4775-a9ef-260b8216c7a4\") " pod="openshift-dns/dns-default-xlzdp" Jul 22 20:06:23.895528 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:23.895509 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zbn42\"" Jul 22 20:06:23.904270 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:23.904250 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xlzdp" Jul 22 20:06:23.989990 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:23.989948 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:24.021032 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:24.020970 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xlzdp"] Jul 22 20:06:24.023825 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:06:24.023802 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01377ab_7b6f_4775_a9ef_260b8216c7a4.slice/crio-ed15cff90b5d12cb832c59dbc3cde4312e2ef2215f0973a43801bf2a2f71867c WatchSource:0}: Error finding container ed15cff90b5d12cb832c59dbc3cde4312e2ef2215f0973a43801bf2a2f71867c: Status 404 returned error can't find the container with id ed15cff90b5d12cb832c59dbc3cde4312e2ef2215f0973a43801bf2a2f71867c Jul 22 20:06:24.108436 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:24.108407 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:24.906319 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:24.906272 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xlzdp" event={"ID":"a01377ab-7b6f-4775-a9ef-260b8216c7a4","Type":"ContainerStarted","Data":"ed15cff90b5d12cb832c59dbc3cde4312e2ef2215f0973a43801bf2a2f71867c"} Jul 22 20:06:26.915073 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:26.915043 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xlzdp" event={"ID":"a01377ab-7b6f-4775-a9ef-260b8216c7a4","Type":"ContainerStarted","Data":"76d47bdaf9ee66544cc5a97846d0469f12739b2c9464e555a0050b41f837dfb2"} Jul 22 20:06:26.915422 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:26.915094 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xlzdp" event={"ID":"a01377ab-7b6f-4775-a9ef-260b8216c7a4","Type":"ContainerStarted","Data":"f8069e1db62a6e4b083dd03b9c8e2064d3cc3280c091cc2d965de74238b295e9"} Jul 22 20:06:26.915422 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:26.915174 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xlzdp" Jul 22 20:06:26.931834 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:26.931788 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xlzdp" podStartSLOduration=252.063670494 podStartE2EDuration="4m13.931774924s" totalImagesPullingTime="1.86810443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:06:26.930580734 +0000 UTC m=+287.443460462" watchObservedRunningTime="2026-07-22 20:06:26.931774924 +0000 UTC m=+287.444654629" Jul 22 20:06:27.896835 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:27.896803 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:27.999267 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:27.999238 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:28.071602 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:28.071570 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:28.217102 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:28.217068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:06:28.219347 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:28.219326 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/677263c5-5fe3-40de-9004-02bb5bebf54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btl22\" (UID: \"677263c5-5fe3-40de-9004-02bb5bebf54b\") " pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:06:28.436326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:28.436296 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-btl22" Jul 22 20:06:28.557180 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:28.557146 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-btl22"] Jul 22 20:06:28.559934 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:06:28.559913 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677263c5_5fe3_40de_9004_02bb5bebf54b.slice/crio-29842eea41ee4400868d279c835d0fe217b63f90d90d57927322328ce18a0ddf WatchSource:0}: Error finding container 29842eea41ee4400868d279c835d0fe217b63f90d90d57927322328ce18a0ddf: Status 404 returned error can't find the container with id 29842eea41ee4400868d279c835d0fe217b63f90d90d57927322328ce18a0ddf Jul 22 20:06:28.716133 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:28.716101 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:28.921744 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:28.921656 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-btl22" event={"ID":"677263c5-5fe3-40de-9004-02bb5bebf54b","Type":"ContainerStarted","Data":"29842eea41ee4400868d279c835d0fe217b63f90d90d57927322328ce18a0ddf"} Jul 22 20:06:31.001864 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:06:31.001817 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-29ngf" podUID="d77c3889-b5fa-491e-b122-3b1c06ac4bd0" Jul 22 20:06:33.079510 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:33.079472 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:33.160054 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:33.160010 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:33.546918 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:33.546877 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:33.646793 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:33.646768 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:33.940891 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:33.940857 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-btl22" event={"ID":"677263c5-5fe3-40de-9004-02bb5bebf54b","Type":"ContainerStarted","Data":"20ae0c9f4b42f047bc7ba51f2b8f31e3d7ff3c77a5a9f415fd0ca7ea89f19647"} Jul 22 20:06:34.056220 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:34.056187 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:34.124750 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:34.124717 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:35.585877 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:35.585793 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:35.686127 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:35.686095 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:36.920725 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:36.920688 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xlzdp" Jul 22 20:06:37.346883 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:37.346852 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:37.958643 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:37.958596 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-btl22" event={"ID":"677263c5-5fe3-40de-9004-02bb5bebf54b","Type":"ContainerStarted","Data":"f2f030f80a854833118ca56da1b90f87796777bc2bcc6904ae1f7521cc3f5747"} Jul 22 20:06:38.935355 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:38.935312 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:39.218605 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:39.218572 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:39.320863 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:39.319731 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:40.227178 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.227145 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:40.340515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.340449 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:40.455638 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.455593 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:40.511707 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.511633 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6q5qz"] Jul 22 20:06:40.524419 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.524374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.526645 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.526620 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 20:06:40.526919 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.526900 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-krxkr\"" Jul 22 20:06:40.527102 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.527082 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 20:06:40.527511 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.527490 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 20:06:40.527603 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.527544 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 20:06:40.527603 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.527490 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 20:06:40.527741 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.527489 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 20:06:40.561047 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.561021 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:40.619561 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.619526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de9b3c28-175e-47ef-9138-4f455e701503-sys\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.619561 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.619565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-tls\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.619788 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.619591 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-wtmp\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.619788 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.619648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-textfile\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.619788 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.619687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de9b3c28-175e-47ef-9138-4f455e701503-metrics-client-ca\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.619788 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.619719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.619788 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.619742 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkn7c\" (UniqueName: \"kubernetes.io/projected/de9b3c28-175e-47ef-9138-4f455e701503-kube-api-access-dkn7c\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.620005 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.619861 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de9b3c28-175e-47ef-9138-4f455e701503-root\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.620005 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.619902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-accelerators-collector-config\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.631883 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.631853 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:40.692385 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.692359 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:06:40.692752 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.692734 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:06:40.694781 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.694761 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 20:06:40.720661 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.720629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de9b3c28-175e-47ef-9138-4f455e701503-root\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.720661 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.720666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-accelerators-collector-config\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.720874 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.720695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de9b3c28-175e-47ef-9138-4f455e701503-sys\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.720874 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.720744 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de9b3c28-175e-47ef-9138-4f455e701503-sys\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.720874 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.720763 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de9b3c28-175e-47ef-9138-4f455e701503-root\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.720874 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.720812 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-tls\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.720874 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.720848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-wtmp\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.720874 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.720870 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-textfile\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.721204 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.720886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de9b3c28-175e-47ef-9138-4f455e701503-metrics-client-ca\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.721204 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.720933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.721204 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.720969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkn7c\" (UniqueName: \"kubernetes.io/projected/de9b3c28-175e-47ef-9138-4f455e701503-kube-api-access-dkn7c\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.721204 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.721010 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-wtmp\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.721204 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.721191 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-textfile\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.721461 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.721275 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-accelerators-collector-config\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.721520 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.721510 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de9b3c28-175e-47ef-9138-4f455e701503-metrics-client-ca\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.723489 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.723462 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.723836 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.723813 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de9b3c28-175e-47ef-9138-4f455e701503-node-exporter-tls\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.732407 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.732386 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkn7c\" (UniqueName: \"kubernetes.io/projected/de9b3c28-175e-47ef-9138-4f455e701503-kube-api-access-dkn7c\") pod \"node-exporter-6q5qz\" (UID: \"de9b3c28-175e-47ef-9138-4f455e701503\") " pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.779630 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.779606 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:40.835871 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.835830 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6q5qz" Jul 22 20:06:40.846411 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:06:40.846386 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9b3c28_175e_47ef_9138_4f455e701503.slice/crio-cc1c86044b94ff33e8505b754f6e88f09a9b7b90465868ef551c5ad29196a158 WatchSource:0}: Error finding container cc1c86044b94ff33e8505b754f6e88f09a9b7b90465868ef551c5ad29196a158: Status 404 returned error can't find the container with id cc1c86044b94ff33e8505b754f6e88f09a9b7b90465868ef551c5ad29196a158 Jul 22 20:06:40.872569 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.872540 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:40.972590 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.972556 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-btl22" event={"ID":"677263c5-5fe3-40de-9004-02bb5bebf54b","Type":"ContainerStarted","Data":"5548fa9d2570717fd930bd59f1e1549ec8ab089badd285246baa1fced7f10ace"} Jul 22 20:06:40.973695 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.973675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6q5qz" event={"ID":"de9b3c28-175e-47ef-9138-4f455e701503","Type":"ContainerStarted","Data":"cc1c86044b94ff33e8505b754f6e88f09a9b7b90465868ef551c5ad29196a158"} Jul 22 20:06:40.991262 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:40.991206 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-btl22" podStartSLOduration=32.924578452 podStartE2EDuration="44.991188096s" totalImagesPullingTime="12.066609644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 20:05:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:06:40.990228378 +0000 UTC m=+301.503108083" watchObservedRunningTime="2026-07-22 20:06:40.991188096 +0000 UTC m=+301.504067802" Jul 22 20:06:41.014021 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:41.013968 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:41.368355 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:41.368321 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:41.485765 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:41.485729 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:41.593353 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:41.593310 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:41.786897 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:41.786861 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:41.883294 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:41.883264 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:41.900643 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:41.900621 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b478545d6-q76ss" Jul 22 20:06:41.978924 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:41.978890 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6q5qz" event={"ID":"de9b3c28-175e-47ef-9138-4f455e701503","Type":"ContainerStarted","Data":"289807c91ad7d3b5ae2d52cd351471837d985512929104ff296b36a4eec85757"} Jul 22 20:06:42.001550 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:42.001519 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:06:42.027353 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:42.027329 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:42.125347 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:42.125247 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:42.983783 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:42.983748 2575 generic.go:356] "Generic (PLEG): container finished" podID="de9b3c28-175e-47ef-9138-4f455e701503" containerID="289807c91ad7d3b5ae2d52cd351471837d985512929104ff296b36a4eec85757" exitCode=0 Jul 22 20:06:42.984206 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:42.983833 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6q5qz" event={"ID":"de9b3c28-175e-47ef-9138-4f455e701503","Type":"ContainerDied","Data":"289807c91ad7d3b5ae2d52cd351471837d985512929104ff296b36a4eec85757"} Jul 22 20:06:43.048431 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:43.048403 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:43.149658 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:43.149627 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:43.270106 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:43.270032 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:43.497196 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:43.497161 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:43.599292 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:43.599211 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:43.783318 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:43.783285 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:43.885674 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:43.885574 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:43.986131 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:43.986101 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:43.988949 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:43.988924 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6q5qz" event={"ID":"de9b3c28-175e-47ef-9138-4f455e701503","Type":"ContainerStarted","Data":"d0762a10fcbcc3deb39669b629723fa6d7856b889d6121c00954aaf4aab797db"} Jul 22 20:06:43.989108 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:43.988998 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6q5qz" event={"ID":"de9b3c28-175e-47ef-9138-4f455e701503","Type":"ContainerStarted","Data":"d3016c1f5e5e816885c459a75932754941b1a4402cd1724ad583e3dc42eed218"} Jul 22 20:06:44.015724 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:44.015674 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6q5qz" podStartSLOduration=2.911094345 podStartE2EDuration="4.015660609s" totalImagesPullingTime="1.040072942s" totalInitContainerRuntime="64.493322ms" isStatefulPod=true podCreationTimestamp="2026-07-22 20:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:06:44.014005718 +0000 UTC m=+304.526885421" watchObservedRunningTime="2026-07-22 20:06:44.015660609 +0000 UTC m=+304.528540314" Jul 22 20:06:44.100769 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:44.100736 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:44.894991 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:44.894954 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f7f6d6f69-x4vql"] Jul 22 20:06:44.898141 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:44.898121 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:44.900504 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:44.900487 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 22 20:06:44.900595 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:44.900514 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-be2rtigsm08rc\"" Jul 22 20:06:44.901402 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:44.901380 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-242lv\"" Jul 22 20:06:44.901519 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:44.901443 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 22 20:06:44.901519 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:44.901445 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 20:06:44.901519 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:44.901445 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 22 20:06:44.910524 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:44.910502 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f7f6d6f69-x4vql"] Jul 22 20:06:45.054349 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.054311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/431ab348-975c-4e0d-9a5a-d45eaf8f92df-audit-log\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.054810 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.054381 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/431ab348-975c-4e0d-9a5a-d45eaf8f92df-secret-metrics-server-tls\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.054810 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.054450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/431ab348-975c-4e0d-9a5a-d45eaf8f92df-metrics-server-audit-profiles\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.054810 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.054494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tplfh\" (UniqueName: \"kubernetes.io/projected/431ab348-975c-4e0d-9a5a-d45eaf8f92df-kube-api-access-tplfh\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.054810 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.054532 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/431ab348-975c-4e0d-9a5a-d45eaf8f92df-client-ca-bundle\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.054810 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.054558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/431ab348-975c-4e0d-9a5a-d45eaf8f92df-secret-metrics-server-client-certs\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.054810 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.054609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431ab348-975c-4e0d-9a5a-d45eaf8f92df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.155166 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.155070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431ab348-975c-4e0d-9a5a-d45eaf8f92df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.155166 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.155110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/431ab348-975c-4e0d-9a5a-d45eaf8f92df-audit-log\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.155166 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.155143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/431ab348-975c-4e0d-9a5a-d45eaf8f92df-secret-metrics-server-tls\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.155166 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.155171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/431ab348-975c-4e0d-9a5a-d45eaf8f92df-metrics-server-audit-profiles\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.155467 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.155192 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tplfh\" (UniqueName: \"kubernetes.io/projected/431ab348-975c-4e0d-9a5a-d45eaf8f92df-kube-api-access-tplfh\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.155467 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.155213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/431ab348-975c-4e0d-9a5a-d45eaf8f92df-client-ca-bundle\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.155467 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.155231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/431ab348-975c-4e0d-9a5a-d45eaf8f92df-secret-metrics-server-client-certs\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.155600 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.155516 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/431ab348-975c-4e0d-9a5a-d45eaf8f92df-audit-log\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.155968 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.155927 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431ab348-975c-4e0d-9a5a-d45eaf8f92df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.156270 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.156247 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/431ab348-975c-4e0d-9a5a-d45eaf8f92df-metrics-server-audit-profiles\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.157740 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.157713 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/431ab348-975c-4e0d-9a5a-d45eaf8f92df-client-ca-bundle\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.157822 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.157744 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/431ab348-975c-4e0d-9a5a-d45eaf8f92df-secret-metrics-server-client-certs\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.157866 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.157829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/431ab348-975c-4e0d-9a5a-d45eaf8f92df-secret-metrics-server-tls\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.163409 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.163387 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tplfh\" (UniqueName: \"kubernetes.io/projected/431ab348-975c-4e0d-9a5a-d45eaf8f92df-kube-api-access-tplfh\") pod \"metrics-server-f7f6d6f69-x4vql\" (UID: \"431ab348-975c-4e0d-9a5a-d45eaf8f92df\") " pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.206926 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.206899 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:06:45.269057 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.269019 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:45.333576 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.333546 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f7f6d6f69-x4vql"] Jul 22 20:06:45.336859 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:06:45.336835 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431ab348_975c_4e0d_9a5a_d45eaf8f92df.slice/crio-d49f69bfba0b7b7fd1a2fd0ec3a2745c2e7cc86f8392feede2363ced58c533ba WatchSource:0}: Error finding container d49f69bfba0b7b7fd1a2fd0ec3a2745c2e7cc86f8392feede2363ced58c533ba: Status 404 returned error can't find the container with id d49f69bfba0b7b7fd1a2fd0ec3a2745c2e7cc86f8392feede2363ced58c533ba Jul 22 20:06:45.337118 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.337093 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:45.338997 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.338965 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 20:06:45.643517 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.643486 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:45.816537 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.816495 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" podUID="1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 22 20:06:45.995831 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:45.995796 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" event={"ID":"431ab348-975c-4e0d-9a5a-d45eaf8f92df","Type":"ContainerStarted","Data":"d49f69bfba0b7b7fd1a2fd0ec3a2745c2e7cc86f8392feede2363ced58c533ba"} Jul 22 20:06:46.532333 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:46.532291 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:46.627357 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:46.627321 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:46.771319 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:46.771282 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:47.000945 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:47.000906 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" event={"ID":"431ab348-975c-4e0d-9a5a-d45eaf8f92df","Type":"ContainerStarted","Data":"26e12bb88080f327021ede402f9026b5ddbbe23b0dd8001396832e4d4dc25b36"} Jul 22 20:06:47.018641 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:47.018582 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" podStartSLOduration=1.484950357 podStartE2EDuration="3.018567873s" totalImagesPullingTime="1.533617516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:06:47.018033627 +0000 UTC m=+307.530913334" watchObservedRunningTime="2026-07-22 20:06:47.018567873 +0000 UTC m=+307.531447557" Jul 22 20:06:48.023089 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:48.023056 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:48.072672 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:48.072638 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:50.889700 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:50.889653 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:50.991310 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:50.991277 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:51.494762 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:51.494729 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:51.571272 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:51.571233 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:55.816455 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:55.816407 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" podUID="1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 22 20:06:56.168705 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:56.168621 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:56.579361 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:56.579319 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:57.908560 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:57.908525 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:06:57.968195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:06:57.968151 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:01.367583 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:01.367548 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:01.468315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:01.468281 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:01.569670 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:01.569624 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:05.207649 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:05.207613 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:07:05.208022 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:05.207682 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:07:05.816677 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:05.816643 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" podUID="1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 22 20:07:05.816821 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:05.816706 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" Jul 22 20:07:05.817203 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:05.817174 2575 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"7b65d9f411aff6d7fbed87ea55e5a7515ce11edbef08c8b0724ec89e10cfbb92"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 22 20:07:05.817247 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:05.817224 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" podUID="1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2" containerName="service-proxy" containerID="cri-o://7b65d9f411aff6d7fbed87ea55e5a7515ce11edbef08c8b0724ec89e10cfbb92" gracePeriod=30 Jul 22 20:07:06.064997 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:06.064897 2575 generic.go:356] "Generic (PLEG): container finished" podID="1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2" containerID="7b65d9f411aff6d7fbed87ea55e5a7515ce11edbef08c8b0724ec89e10cfbb92" exitCode=2 Jul 22 20:07:06.064997 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:06.064962 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" event={"ID":"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2","Type":"ContainerDied","Data":"7b65d9f411aff6d7fbed87ea55e5a7515ce11edbef08c8b0724ec89e10cfbb92"} Jul 22 20:07:06.065171 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:06.065027 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c874bfb96-8cqpx" event={"ID":"1d081a90-1e14-4bcd-92e7-0ce9dbbf14c2","Type":"ContainerStarted","Data":"e5f10eeb52b94b6a1ab2b9e60b0bd3f806bb3089df309ae0b86ddd3c457e1caa"} Jul 22 20:07:13.012406 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:13.012375 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:13.107197 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:13.107164 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:25.213214 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:25.213187 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:07:25.216906 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:25.216885 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f7f6d6f69-x4vql" Jul 22 20:07:28.022653 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:28.022622 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:28.106533 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:28.106502 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:47.475732 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:47.475648 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:47.534219 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:47.534158 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:52.879239 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:52.879195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:07:52.881449 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:52.881426 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77c3889-b5fa-491e-b122-3b1c06ac4bd0-metrics-certs\") pod \"network-metrics-daemon-29ngf\" (UID: \"d77c3889-b5fa-491e-b122-3b1c06ac4bd0\") " pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:07:53.104882 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:53.104846 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l85mx\"" Jul 22 20:07:53.113044 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:53.113016 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29ngf" Jul 22 20:07:53.240905 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:53.240871 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29ngf"] Jul 22 20:07:53.244166 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:07:53.244139 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77c3889_b5fa_491e_b122_3b1c06ac4bd0.slice/crio-c846a68085385c126740b981f1d94b09a6a2f4725cdd739b1b6c83f870245c05 WatchSource:0}: Error finding container c846a68085385c126740b981f1d94b09a6a2f4725cdd739b1b6c83f870245c05: Status 404 returned error can't find the container with id c846a68085385c126740b981f1d94b09a6a2f4725cdd739b1b6c83f870245c05 Jul 22 20:07:54.223188 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:54.223149 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29ngf" event={"ID":"d77c3889-b5fa-491e-b122-3b1c06ac4bd0","Type":"ContainerStarted","Data":"c846a68085385c126740b981f1d94b09a6a2f4725cdd739b1b6c83f870245c05"} Jul 22 20:07:55.227965 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:55.227932 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29ngf" event={"ID":"d77c3889-b5fa-491e-b122-3b1c06ac4bd0","Type":"ContainerStarted","Data":"387a384acd38f54d9d9d524cc3a17df385df043a98abad8a4d02fafa5c4af4a4"} Jul 22 20:07:55.228374 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:55.227998 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29ngf" event={"ID":"d77c3889-b5fa-491e-b122-3b1c06ac4bd0","Type":"ContainerStarted","Data":"5ff71fce5d6d3d37a30e3190aaaa0bdd257b3fc49d0d2174a3e4343e47ad2d52"} Jul 22 20:07:55.243822 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:55.243772 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-29ngf" podStartSLOduration=373.757648788 podStartE2EDuration="6m15.243756528s" totalImagesPullingTime="1.48610774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:07:55.242364555 +0000 UTC m=+375.755244262" watchObservedRunningTime="2026-07-22 20:07:55.243756528 +0000 UTC m=+375.756636296" Jul 22 20:07:55.327996 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:55.327950 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:55.390122 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:55.390093 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:56.634086 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:56.634055 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:56.810107 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:56.810065 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:58.398527 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:58.398489 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:58.498920 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:58.498884 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:58.599815 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:58.599761 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:59.138950 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:59.138918 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:07:59.191515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:07:59.191489 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:03.172558 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:03.172527 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:03.273430 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:03.273368 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:08.570998 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:08.570946 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:08.637271 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:08.637240 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:09.617010 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:09.616964 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:09.716039 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:09.716009 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:09.774172 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:09.774140 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:13.001694 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:13.001659 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:13.112402 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:13.112374 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:32.150836 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:32.150792 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:32.250393 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:32.250357 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:33.207331 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:33.207297 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:45.470285 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:45.470251 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:08:45.567507 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:08:45.567472 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:13.176471 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:13.176431 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:13.837570 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:13.837530 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:16.045749 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:16.045720 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:16.151396 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:16.151366 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:17.765354 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:17.765325 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:17.876839 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:17.876803 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:18.225423 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:18.225392 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:18.412067 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:18.412025 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:20.177255 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:20.177217 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:20.404990 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:20.404933 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:23.038248 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:23.038214 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:23.139582 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:23.139554 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:27.377148 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:27.377121 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:32.430102 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:32.430069 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:32.674147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:32.674114 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:39.246149 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:39.246113 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:39.346139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:39.346107 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:40.046752 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:40.046714 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:40.831256 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:40.831225 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:41.392795 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:41.392761 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:41.461410 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:41.461384 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:55.115940 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.115903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:55.186439 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.186403 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf"] Jul 22 20:09:55.189419 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.189402 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.191824 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.191801 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 20:09:55.192710 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.192677 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 20:09:55.192710 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.192698 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mzp9p\"" Jul 22 20:09:55.192710 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.192704 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 20:09:55.192931 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.192707 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 20:09:55.192931 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.192788 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 20:09:55.201421 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.201396 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf"] Jul 22 20:09:55.211698 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.211668 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:55.324938 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.324890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bc578c9-18bc-4546-a051-a0ca2516502b-metrics-cert\") pod \"lws-controller-manager-76bfc99684-f8pnf\" (UID: \"1bc578c9-18bc-4546-a051-a0ca2516502b\") " pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.324938 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.324937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1bc578c9-18bc-4546-a051-a0ca2516502b-manager-config\") pod \"lws-controller-manager-76bfc99684-f8pnf\" (UID: \"1bc578c9-18bc-4546-a051-a0ca2516502b\") " pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.325194 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.324954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87t7b\" (UniqueName: \"kubernetes.io/projected/1bc578c9-18bc-4546-a051-a0ca2516502b-kube-api-access-87t7b\") pod \"lws-controller-manager-76bfc99684-f8pnf\" (UID: \"1bc578c9-18bc-4546-a051-a0ca2516502b\") " pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.325194 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.325051 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bc578c9-18bc-4546-a051-a0ca2516502b-cert\") pod \"lws-controller-manager-76bfc99684-f8pnf\" (UID: \"1bc578c9-18bc-4546-a051-a0ca2516502b\") " pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.425939 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.425847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1bc578c9-18bc-4546-a051-a0ca2516502b-manager-config\") pod \"lws-controller-manager-76bfc99684-f8pnf\" (UID: \"1bc578c9-18bc-4546-a051-a0ca2516502b\") " pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.425939 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.425889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87t7b\" (UniqueName: \"kubernetes.io/projected/1bc578c9-18bc-4546-a051-a0ca2516502b-kube-api-access-87t7b\") pod \"lws-controller-manager-76bfc99684-f8pnf\" (UID: \"1bc578c9-18bc-4546-a051-a0ca2516502b\") " pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.426180 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.426011 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bc578c9-18bc-4546-a051-a0ca2516502b-cert\") pod \"lws-controller-manager-76bfc99684-f8pnf\" (UID: \"1bc578c9-18bc-4546-a051-a0ca2516502b\") " pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.426180 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.426077 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bc578c9-18bc-4546-a051-a0ca2516502b-metrics-cert\") pod \"lws-controller-manager-76bfc99684-f8pnf\" (UID: \"1bc578c9-18bc-4546-a051-a0ca2516502b\") " pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.426598 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.426578 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1bc578c9-18bc-4546-a051-a0ca2516502b-manager-config\") pod \"lws-controller-manager-76bfc99684-f8pnf\" (UID: \"1bc578c9-18bc-4546-a051-a0ca2516502b\") " pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.428446 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.428424 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bc578c9-18bc-4546-a051-a0ca2516502b-cert\") pod \"lws-controller-manager-76bfc99684-f8pnf\" (UID: \"1bc578c9-18bc-4546-a051-a0ca2516502b\") " pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.428528 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.428460 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bc578c9-18bc-4546-a051-a0ca2516502b-metrics-cert\") pod \"lws-controller-manager-76bfc99684-f8pnf\" (UID: \"1bc578c9-18bc-4546-a051-a0ca2516502b\") " pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.444299 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.444269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87t7b\" (UniqueName: \"kubernetes.io/projected/1bc578c9-18bc-4546-a051-a0ca2516502b-kube-api-access-87t7b\") pod \"lws-controller-manager-76bfc99684-f8pnf\" (UID: \"1bc578c9-18bc-4546-a051-a0ca2516502b\") " pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.498857 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.498808 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:55.631201 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:55.631155 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf"] Jul 22 20:09:55.633904 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:09:55.633877 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc578c9_18bc_4546_a051_a0ca2516502b.slice/crio-aa9be80f92897832a3f7a845577ea8ea484465151c73bcf148024dc0b529cc4e WatchSource:0}: Error finding container aa9be80f92897832a3f7a845577ea8ea484465151c73bcf148024dc0b529cc4e: Status 404 returned error can't find the container with id aa9be80f92897832a3f7a845577ea8ea484465151c73bcf148024dc0b529cc4e Jul 22 20:09:56.622836 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:56.622807 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" event={"ID":"1bc578c9-18bc-4546-a051-a0ca2516502b","Type":"ContainerStarted","Data":"aa9be80f92897832a3f7a845577ea8ea484465151c73bcf148024dc0b529cc4e"} Jul 22 20:09:57.092430 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:57.092392 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:57.190626 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:57.190586 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:57.647333 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:57.647291 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:57.873833 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:57.873798 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:57.974671 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:57.974633 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:58.019292 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.019258 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl"] Jul 22 20:09:58.024181 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.024160 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.027964 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.027937 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 20:09:58.027964 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.027952 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 20:09:58.028154 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.027938 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 20:09:58.029417 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.028789 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 20:09:58.029417 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.029075 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zqsx9\"" Jul 22 20:09:58.056686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.056653 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl"] Jul 22 20:09:58.146449 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.146412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97c53fcf-3ae1-49af-a341-42aab4dfba86-tmp\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.146449 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.146450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97c53fcf-3ae1-49af-a341-42aab4dfba86-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.146693 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.146570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97c53fcf-3ae1-49af-a341-42aab4dfba86-webhook-cert\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.146693 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.146658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d556\" (UniqueName: \"kubernetes.io/projected/97c53fcf-3ae1-49af-a341-42aab4dfba86-kube-api-access-7d556\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.146796 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.146695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97c53fcf-3ae1-49af-a341-42aab4dfba86-manifests\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.206628 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.206592 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:58.247859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.247776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d556\" (UniqueName: \"kubernetes.io/projected/97c53fcf-3ae1-49af-a341-42aab4dfba86-kube-api-access-7d556\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.247859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.247819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97c53fcf-3ae1-49af-a341-42aab4dfba86-manifests\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.247859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.247852 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97c53fcf-3ae1-49af-a341-42aab4dfba86-tmp\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.248174 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.247878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97c53fcf-3ae1-49af-a341-42aab4dfba86-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.248174 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.247915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97c53fcf-3ae1-49af-a341-42aab4dfba86-webhook-cert\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.248308 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.248276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97c53fcf-3ae1-49af-a341-42aab4dfba86-manifests\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.250475 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.250450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97c53fcf-3ae1-49af-a341-42aab4dfba86-tmp\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.250573 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.250519 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97c53fcf-3ae1-49af-a341-42aab4dfba86-webhook-cert\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.250631 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.250609 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97c53fcf-3ae1-49af-a341-42aab4dfba86-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.256724 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.256683 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d556\" (UniqueName: \"kubernetes.io/projected/97c53fcf-3ae1-49af-a341-42aab4dfba86-kube-api-access-7d556\") pod \"opendatahub-operator-controller-manager-9fc56f4f-78xpl\" (UID: \"97c53fcf-3ae1-49af-a341-42aab4dfba86\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.311832 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.311786 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:58.337962 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.337921 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:09:58.412244 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.412218 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:58.473904 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.473880 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:58.507440 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.507393 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl"] Jul 22 20:09:58.510263 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:09:58.510243 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c53fcf_3ae1_49af_a341_42aab4dfba86.slice/crio-d3ac4165ba6cb37f3956f15f6ca143493ef92a072ab03c452df4d0d62e7eb322 WatchSource:0}: Error finding container d3ac4165ba6cb37f3956f15f6ca143493ef92a072ab03c452df4d0d62e7eb322: Status 404 returned error can't find the container with id d3ac4165ba6cb37f3956f15f6ca143493ef92a072ab03c452df4d0d62e7eb322 Jul 22 20:09:58.541817 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.541787 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:58.630232 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.630200 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" event={"ID":"97c53fcf-3ae1-49af-a341-42aab4dfba86","Type":"ContainerStarted","Data":"d3ac4165ba6cb37f3956f15f6ca143493ef92a072ab03c452df4d0d62e7eb322"} Jul 22 20:09:58.631663 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.631639 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" event={"ID":"1bc578c9-18bc-4546-a051-a0ca2516502b","Type":"ContainerStarted","Data":"49ba6fffba29338bfb965d3c9f7eba7a164ff2de6acf305b41784dc9e2a40283"} Jul 22 20:09:58.631821 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.631782 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:09:58.636046 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.636025 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:09:58.650136 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.650095 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" podStartSLOduration=0.855611164 podStartE2EDuration="3.650081209s" totalImagesPullingTime="2.794470045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:09:58.648998977 +0000 UTC m=+499.161878684" watchObservedRunningTime="2026-07-22 20:09:58.650081209 +0000 UTC m=+499.162960915" Jul 22 20:09:58.745378 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:09:58.745331 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:02.650125 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:02.650086 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" event={"ID":"97c53fcf-3ae1-49af-a341-42aab4dfba86","Type":"ContainerStarted","Data":"6099f66d16654b530af801ffdcaa2e9e5c83acf45cfb87cb7751d6b9546dbb3e"} Jul 22 20:10:03.654749 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:03.654715 2575 generic.go:356] "Generic (PLEG): container finished" podID="97c53fcf-3ae1-49af-a341-42aab4dfba86" containerID="6099f66d16654b530af801ffdcaa2e9e5c83acf45cfb87cb7751d6b9546dbb3e" exitCode=0 Jul 22 20:10:03.655151 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:03.654774 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" event={"ID":"97c53fcf-3ae1-49af-a341-42aab4dfba86","Type":"ContainerDied","Data":"6099f66d16654b530af801ffdcaa2e9e5c83acf45cfb87cb7751d6b9546dbb3e"} Jul 22 20:10:04.660659 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:04.660620 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" event={"ID":"97c53fcf-3ae1-49af-a341-42aab4dfba86","Type":"ContainerStarted","Data":"489a8da061f440454d95cb9b3d0b7e9ebf8da3a3f9a3beae654ef2ade28a9a0e"} Jul 22 20:10:04.661090 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:04.660746 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:10:04.699986 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:04.699934 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" podStartSLOduration=2.758085151 podStartE2EDuration="7.699917139s" totalImagesPullingTime="3.878333288s" totalInitContainerRuntime="1.0634987s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:09:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:10:04.697701879 +0000 UTC m=+505.210581584" watchObservedRunningTime="2026-07-22 20:10:04.699917139 +0000 UTC m=+505.212796918" Jul 22 20:10:09.638834 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:09.638807 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76bfc99684-f8pnf" Jul 22 20:10:10.012835 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:10.012794 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:10.274884 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:10.274795 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:15.666233 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:15.666201 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9fc56f4f-78xpl" Jul 22 20:10:18.576897 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:18.576864 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:18.754414 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:18.754361 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:20.017386 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:20.017349 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:20.320574 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:20.320471 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:20.640260 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:20.640179 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:20.915814 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:20.915727 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:21.140245 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:21.140212 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:22.289178 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:22.289147 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:22.769923 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:22.769885 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:24.876933 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:24.876900 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:25.065941 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:25.065891 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:28.378580 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:28.378542 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:29.789586 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:29.789532 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:30.061950 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:30.061861 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:31.800314 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.800240 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l"] Jul 22 20:10:31.816048 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.816016 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:31.829672 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.829647 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 20:10:31.831466 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.831446 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gjptv\"" Jul 22 20:10:31.885776 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.885747 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 20:10:31.885943 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.885783 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 20:10:31.898661 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.898636 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:31.898848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.898664 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:31.898848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.898685 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:31.898848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.898703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:31.898848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.898726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:31.898848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.898775 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:31.898848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.898805 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:31.898848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.898838 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:31.899120 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.898874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chfcz\" (UniqueName: \"kubernetes.io/projected/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-kube-api-access-chfcz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:31.945422 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.945387 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:31.999790 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.999758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:31.999790 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.999792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chfcz\" (UniqueName: \"kubernetes.io/projected/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-kube-api-access-chfcz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.000074 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.999926 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.000074 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.999953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.000074 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:31.999992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.000074 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.000021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.000074 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.000041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.000312 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.000092 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.000312 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.000134 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.000409 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.000385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.000520 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.000498 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.000576 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.000527 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.000721 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.000699 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.000818 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.000802 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.002226 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.002203 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.002542 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.002522 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.016657 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.016617 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chfcz\" (UniqueName: \"kubernetes.io/projected/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-kube-api-access-chfcz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.055136 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.055059 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l"] Jul 22 20:10:32.076444 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.076410 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0c6221-5afb-4a54-8ee5-c856d3a0e77d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46482l\" (UID: \"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.125766 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.125726 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:32.234378 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.234346 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r"] Jul 22 20:10:32.237826 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.237795 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.246528 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.246500 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 20:10:32.246528 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.246500 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 20:10:32.246719 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:10:32.246581 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" is forbidden: User \"system:node:ip-10-0-143-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-143-196.ec2.internal' and this object" podUID="cb12e370-8f88-49af-8643-ccffb2f996a6" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.247501 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:10:32.247474 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"istiod-openshift-gateway-dockercfg-6ztjs\" is forbidden: User \"system:node:ip-10-0-143-196.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-143-196.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6ztjs\"" type="*v1.Secret" Jul 22 20:10:32.247902 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.247887 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:32.250411 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.250394 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 20:10:32.299570 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.299526 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r"] Jul 22 20:10:32.302358 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.302328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb12e370-8f88-49af-8643-ccffb2f996a6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.302511 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.302376 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cb12e370-8f88-49af-8643-ccffb2f996a6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.302511 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.302396 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb12e370-8f88-49af-8643-ccffb2f996a6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.302511 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.302471 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cb12e370-8f88-49af-8643-ccffb2f996a6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.302668 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.302512 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cb12e370-8f88-49af-8643-ccffb2f996a6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.302668 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.302536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb12e370-8f88-49af-8643-ccffb2f996a6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.302668 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.302625 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxtmp\" (UniqueName: \"kubernetes.io/projected/cb12e370-8f88-49af-8643-ccffb2f996a6-kube-api-access-hxtmp\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.375551 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.373276 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l"] Jul 22 20:10:32.403682 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.403641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb12e370-8f88-49af-8643-ccffb2f996a6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.403848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.403699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cb12e370-8f88-49af-8643-ccffb2f996a6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.403848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.403717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb12e370-8f88-49af-8643-ccffb2f996a6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.403848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.403734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cb12e370-8f88-49af-8643-ccffb2f996a6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.403848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.403757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cb12e370-8f88-49af-8643-ccffb2f996a6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.403848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.403775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb12e370-8f88-49af-8643-ccffb2f996a6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.404134 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.403930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxtmp\" (UniqueName: \"kubernetes.io/projected/cb12e370-8f88-49af-8643-ccffb2f996a6-kube-api-access-hxtmp\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.404648 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.404622 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cb12e370-8f88-49af-8643-ccffb2f996a6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.406183 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.406163 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cb12e370-8f88-49af-8643-ccffb2f996a6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.406274 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.406208 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cb12e370-8f88-49af-8643-ccffb2f996a6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.406418 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.406397 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb12e370-8f88-49af-8643-ccffb2f996a6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.406521 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.406505 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb12e370-8f88-49af-8643-ccffb2f996a6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.431097 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.431063 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxtmp\" (UniqueName: \"kubernetes.io/projected/cb12e370-8f88-49af-8643-ccffb2f996a6-kube-api-access-hxtmp\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.441906 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.441878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb12e370-8f88-49af-8643-ccffb2f996a6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5nc2r\" (UID: \"cb12e370-8f88-49af-8643-ccffb2f996a6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:32.595499 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.595468 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:32.761890 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:32.761852 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" event={"ID":"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d","Type":"ContainerStarted","Data":"d5df8658855e33212f920b36b8fb6694d9d1cfa18054044a9005e20057087b8d"} Jul 22 20:10:33.488863 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:33.488833 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6ztjs\"" Jul 22 20:10:33.497832 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:33.497804 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:33.694579 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:33.694542 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r"] Jul 22 20:10:33.699630 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:10:33.699593 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb12e370_8f88_49af_8643_ccffb2f996a6.slice/crio-e98639e64752490b4196cd95e05cc393db7fe9a811d8d45de24a1ddac4ac40cc WatchSource:0}: Error finding container e98639e64752490b4196cd95e05cc393db7fe9a811d8d45de24a1ddac4ac40cc: Status 404 returned error can't find the container with id e98639e64752490b4196cd95e05cc393db7fe9a811d8d45de24a1ddac4ac40cc Jul 22 20:10:33.767440 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:33.767353 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" event={"ID":"cb12e370-8f88-49af-8643-ccffb2f996a6","Type":"ContainerStarted","Data":"e98639e64752490b4196cd95e05cc393db7fe9a811d8d45de24a1ddac4ac40cc"} Jul 22 20:10:35.045767 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:35.045726 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:10:35.046073 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:35.045800 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:10:35.046073 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:35.045830 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:10:35.124896 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:35.124870 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:35.431015 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:35.430913 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:35.702098 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:35.702062 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:35.777163 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:35.777123 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" event={"ID":"cf0c6221-5afb-4a54-8ee5-c856d3a0e77d","Type":"ContainerStarted","Data":"cb1c5078bc714f6b2bce68a8e9e620bb09ab91cad4460a452b668fbb148c5421"} Jul 22 20:10:35.920533 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:35.920468 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" podStartSLOduration=2.252821234 podStartE2EDuration="4.920447933s" totalImagesPullingTime="2.667626699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:10:35.817263106 +0000 UTC m=+536.330142810" watchObservedRunningTime="2026-07-22 20:10:35.920447933 +0000 UTC m=+536.433327639" Jul 22 20:10:36.033835 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:36.033747 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:36.126825 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:36.126785 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:36.132138 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:36.132113 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:36.781910 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:36.781881 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:36.783276 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:36.783253 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46482l" Jul 22 20:10:36.987315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:36.987273 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:10:36.987424 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:36.987369 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:10:37.398159 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:37.398065 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:37.591500 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:37.591462 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:37.787277 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:37.787236 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" event={"ID":"cb12e370-8f88-49af-8643-ccffb2f996a6","Type":"ContainerStarted","Data":"c1c8b9217e3b81ec7e91c51a7190af2ba2f0433ffe4956bb71603a2d299fc31c"} Jul 22 20:10:37.860567 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:37.860533 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:10:38.012326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:38.012259 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" podStartSLOduration=2.727285698 podStartE2EDuration="6.012241486s" totalImagesPullingTime="3.284955788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:10:38.0119007 +0000 UTC m=+538.524780408" watchObservedRunningTime="2026-07-22 20:10:38.012241486 +0000 UTC m=+538.525121190" Jul 22 20:10:38.791422 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:38.791392 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:38.792963 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:38.792944 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5nc2r" Jul 22 20:10:50.039743 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:10:50.039707 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:20.427355 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:20.427309 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:20.642439 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:20.642406 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:20.776085 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:20.776052 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:35.293098 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:35.293059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:35.387524 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:35.387488 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:37.417605 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:37.417572 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:40.713718 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:40.713690 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:11:40.714918 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:40.713759 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:11:50.886352 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:50.886290 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:53.655706 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:53.655669 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:54.920993 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:54.920951 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:55.060965 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:55.060929 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:56.345072 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:56.345036 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:57.256906 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:57.256836 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:59.171307 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:59.171272 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:11:59.311116 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:11:59.311085 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:01.911251 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:01.911175 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:02.334702 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:02.334669 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:04.869706 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:04.869676 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:06.159381 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:06.159297 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:06.319632 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:06.319602 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:06.420965 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:06.420885 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:07.948145 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:07.948113 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:08.043164 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:08.043134 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:11.404949 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:11.404918 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:11.473094 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:11.473065 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:15.879169 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:15.879133 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:15.980887 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:15.980852 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:17.978526 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:17.978493 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:18.096099 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:18.096066 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:19.003684 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:19.003651 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:19.382300 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:19.382223 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:27.127411 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:27.127374 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:27.548917 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:27.548883 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:27.676333 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:27.676293 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:27.778488 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:27.778459 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:30.681971 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:30.681938 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:30.833541 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:30.833508 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:35.222513 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:35.222480 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:35.886760 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:35.886727 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:37.775861 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:37.775831 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:12:37.877538 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:12:37.877505 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:01.657093 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:01.657049 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:01.759781 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:01.759748 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:07.378881 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.378850 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:07.484499 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.484469 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:07.585063 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.585025 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:07.772039 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.772003 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:13:07.775165 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.775147 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcbb2" Jul 22 20:13:07.777441 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.777403 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-x4sxh\"" Jul 22 20:13:07.777554 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.777423 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 20:13:07.778314 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.778296 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 20:13:07.778314 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.778308 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 20:13:07.783013 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.782992 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:13:07.818568 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.818533 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:13:07.872549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.872514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/28678c94-d0ce-4e87-8336-da8bb10eb673-config-file\") pod \"limitador-limitador-54d6d988d5-fcbb2\" (UID: \"28678c94-d0ce-4e87-8336-da8bb10eb673\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcbb2" Jul 22 20:13:07.872689 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.872579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l457\" (UniqueName: \"kubernetes.io/projected/28678c94-d0ce-4e87-8336-da8bb10eb673-kube-api-access-2l457\") pod \"limitador-limitador-54d6d988d5-fcbb2\" (UID: \"28678c94-d0ce-4e87-8336-da8bb10eb673\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcbb2" Jul 22 20:13:07.973202 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.973164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l457\" (UniqueName: \"kubernetes.io/projected/28678c94-d0ce-4e87-8336-da8bb10eb673-kube-api-access-2l457\") pod \"limitador-limitador-54d6d988d5-fcbb2\" (UID: \"28678c94-d0ce-4e87-8336-da8bb10eb673\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcbb2" Jul 22 20:13:07.973397 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.973229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/28678c94-d0ce-4e87-8336-da8bb10eb673-config-file\") pod \"limitador-limitador-54d6d988d5-fcbb2\" (UID: \"28678c94-d0ce-4e87-8336-da8bb10eb673\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcbb2" Jul 22 20:13:07.973855 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.973828 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/28678c94-d0ce-4e87-8336-da8bb10eb673-config-file\") pod \"limitador-limitador-54d6d988d5-fcbb2\" (UID: \"28678c94-d0ce-4e87-8336-da8bb10eb673\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcbb2" Jul 22 20:13:07.980581 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:07.980556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l457\" (UniqueName: \"kubernetes.io/projected/28678c94-d0ce-4e87-8336-da8bb10eb673-kube-api-access-2l457\") pod \"limitador-limitador-54d6d988d5-fcbb2\" (UID: \"28678c94-d0ce-4e87-8336-da8bb10eb673\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcbb2" Jul 22 20:13:08.085933 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:08.085841 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcbb2" Jul 22 20:13:08.216838 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:08.216801 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:13:08.219994 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:13:08.219946 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28678c94_d0ce_4e87_8336_da8bb10eb673.slice/crio-93c550a6a823e7dafe973bd6ba2904984076f20c0a11fbda0223c95e8d950797 WatchSource:0}: Error finding container 93c550a6a823e7dafe973bd6ba2904984076f20c0a11fbda0223c95e8d950797: Status 404 returned error can't find the container with id 93c550a6a823e7dafe973bd6ba2904984076f20c0a11fbda0223c95e8d950797 Jul 22 20:13:08.221853 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:08.221837 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 20:13:08.231824 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:08.231786 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:08.319036 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:08.319004 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcbb2" event={"ID":"28678c94-d0ce-4e87-8336-da8bb10eb673","Type":"ContainerStarted","Data":"93c550a6a823e7dafe973bd6ba2904984076f20c0a11fbda0223c95e8d950797"} Jul 22 20:13:08.333217 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:08.333186 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:11.331817 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:11.331781 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcbb2" event={"ID":"28678c94-d0ce-4e87-8336-da8bb10eb673","Type":"ContainerStarted","Data":"00d4f380fa8598cb9a2ec9d8d5c6d55ac990eac07734a345587ce88c2b94645e"} Jul 22 20:13:11.332180 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:11.331847 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcbb2" Jul 22 20:13:11.348774 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:11.348732 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcbb2" podStartSLOduration=2.021652982 podStartE2EDuration="4.348715872s" totalImagesPullingTime="2.32706289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:13:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:13:11.347002946 +0000 UTC m=+691.859882649" watchObservedRunningTime="2026-07-22 20:13:11.348715872 +0000 UTC m=+691.861595578" Jul 22 20:13:16.864632 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:16.864599 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:16.966006 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:16.965953 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:22.336602 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:22.336571 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcbb2" Jul 22 20:13:23.532368 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:23.532336 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:23.633496 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:23.633450 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:24.213708 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:24.213670 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:36.847590 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:36.847508 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:37.139128 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:37.139031 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:43.404760 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:43.404721 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:43.469578 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:43.469527 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:43.571101 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:43.571069 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:44.134893 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:44.134856 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:47.603306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:47.603270 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:13:47.705658 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:13:47.705619 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:14:01.696162 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:14:01.696125 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:14:22.268280 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:14:22.268243 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:14:22.368424 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:14:22.368384 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:14:22.768915 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:14:22.768886 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:14:22.935404 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:14:22.935368 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:14:23.299465 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:14:23.299433 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:14:23.396037 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:14:23.395999 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:14:23.768852 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:14:23.768819 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:14:33.908328 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:14:33.908289 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:14:34.150188 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:14:34.150151 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:14:45.572163 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:14:45.572114 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:14:46.065859 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:14:46.065825 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:15:08.332049 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:15:08.332013 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:15:09.069281 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:15:09.069237 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:15:10.020435 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:15:10.020399 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:15:24.316452 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:15:24.316414 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:15:25.479234 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:15:25.479201 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:15:48.054821 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:15:48.054784 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:15:49.771956 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:15:49.771917 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:15:51.093366 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:15:51.093328 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:15:51.195284 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:15:51.195254 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:16:02.517850 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:16:02.517808 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:16:04.777936 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:16:04.777902 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:16:21.746682 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:16:21.746643 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:16:21.847183 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:16:21.847149 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:16:34.323364 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:16:34.323272 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:16:35.848501 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:16:35.848468 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:16:38.997395 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:16:38.997358 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:16:40.736694 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:16:40.736587 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:16:40.743294 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:16:40.737896 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:16:59.627339 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:16:59.627304 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:17:01.748366 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:17:01.748335 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:17:18.934432 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:17:18.934393 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:17:19.036423 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:17:19.036392 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:17:20.101538 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:17:20.101490 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:17:20.201028 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:17:20.200998 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:18:03.686391 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:18:03.686306 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:18:15.597274 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:18:15.597236 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:18:24.877681 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:18:24.877643 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:19:16.021712 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:16.021660 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:19:17.810143 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:17.810104 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:19:23.362081 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:23.362041 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64cc9776b5-kr86l"] Jul 22 20:19:23.365260 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:23.365236 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64cc9776b5-kr86l" Jul 22 20:19:23.370292 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:23.370270 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 20:19:23.370424 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:23.370329 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kkclf\"" Jul 22 20:19:23.371448 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:23.371301 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 20:19:23.393407 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:23.393379 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64cc9776b5-kr86l"] Jul 22 20:19:23.412067 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:23.412036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c65d990a-7650-476d-acfb-bc8493ae77d0-webhook-cert\") pod \"maas-controller-64cc9776b5-kr86l\" (UID: \"c65d990a-7650-476d-acfb-bc8493ae77d0\") " pod="opendatahub/maas-controller-64cc9776b5-kr86l" Jul 22 20:19:23.412217 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:23.412081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf4dt\" (UniqueName: \"kubernetes.io/projected/c65d990a-7650-476d-acfb-bc8493ae77d0-kube-api-access-rf4dt\") pod \"maas-controller-64cc9776b5-kr86l\" (UID: \"c65d990a-7650-476d-acfb-bc8493ae77d0\") " pod="opendatahub/maas-controller-64cc9776b5-kr86l" Jul 22 20:19:23.512701 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:23.512666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c65d990a-7650-476d-acfb-bc8493ae77d0-webhook-cert\") pod \"maas-controller-64cc9776b5-kr86l\" (UID: \"c65d990a-7650-476d-acfb-bc8493ae77d0\") " pod="opendatahub/maas-controller-64cc9776b5-kr86l" Jul 22 20:19:23.512892 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:23.512718 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf4dt\" (UniqueName: \"kubernetes.io/projected/c65d990a-7650-476d-acfb-bc8493ae77d0-kube-api-access-rf4dt\") pod \"maas-controller-64cc9776b5-kr86l\" (UID: \"c65d990a-7650-476d-acfb-bc8493ae77d0\") " pod="opendatahub/maas-controller-64cc9776b5-kr86l" Jul 22 20:19:23.515062 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:23.515038 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c65d990a-7650-476d-acfb-bc8493ae77d0-webhook-cert\") pod \"maas-controller-64cc9776b5-kr86l\" (UID: \"c65d990a-7650-476d-acfb-bc8493ae77d0\") " pod="opendatahub/maas-controller-64cc9776b5-kr86l" Jul 22 20:19:23.521338 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:23.521317 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf4dt\" (UniqueName: \"kubernetes.io/projected/c65d990a-7650-476d-acfb-bc8493ae77d0-kube-api-access-rf4dt\") pod \"maas-controller-64cc9776b5-kr86l\" (UID: \"c65d990a-7650-476d-acfb-bc8493ae77d0\") " pod="opendatahub/maas-controller-64cc9776b5-kr86l" Jul 22 20:19:23.674109 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:23.674025 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64cc9776b5-kr86l" Jul 22 20:19:24.010080 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:24.010057 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64cc9776b5-kr86l"] Jul 22 20:19:24.012779 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:19:24.012756 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65d990a_7650_476d_acfb_bc8493ae77d0.slice/crio-acbacbcd7847906aaf8bad6d763181eab45190e92451ab9818250bf84658c3af WatchSource:0}: Error finding container acbacbcd7847906aaf8bad6d763181eab45190e92451ab9818250bf84658c3af: Status 404 returned error can't find the container with id acbacbcd7847906aaf8bad6d763181eab45190e92451ab9818250bf84658c3af Jul 22 20:19:24.014441 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:24.014423 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 20:19:24.674805 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:24.674768 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64cc9776b5-kr86l" event={"ID":"c65d990a-7650-476d-acfb-bc8493ae77d0","Type":"ContainerStarted","Data":"acbacbcd7847906aaf8bad6d763181eab45190e92451ab9818250bf84658c3af"} Jul 22 20:19:26.684579 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:26.684479 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64cc9776b5-kr86l" event={"ID":"c65d990a-7650-476d-acfb-bc8493ae77d0","Type":"ContainerStarted","Data":"6b04dc59f13b9075d6d791c522805e843f4af3074b17b062847864b37c847b21"} Jul 22 20:19:26.684919 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:26.684582 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64cc9776b5-kr86l" Jul 22 20:19:26.718609 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:26.718550 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-64cc9776b5-kr86l" podStartSLOduration=1.361442018 podStartE2EDuration="3.718534641s" totalImagesPullingTime="2.357092623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:19:26.716088446 +0000 UTC m=+1067.228968154" watchObservedRunningTime="2026-07-22 20:19:26.718534641 +0000 UTC m=+1067.231414347" Jul 22 20:19:37.691664 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:37.691590 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64cc9776b5-kr86l" Jul 22 20:19:42.239280 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.239236 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:19:42.333653 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.333615 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64cc9776b5-kr86l"] Jul 22 20:19:42.333864 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.333838 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-64cc9776b5-kr86l" podUID="c65d990a-7650-476d-acfb-bc8493ae77d0" containerName="manager" containerID="cri-o://6b04dc59f13b9075d6d791c522805e843f4af3074b17b062847864b37c847b21" gracePeriod=30 Jul 22 20:19:42.572507 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.572484 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64cc9776b5-kr86l" Jul 22 20:19:42.665218 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.665186 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c65d990a-7650-476d-acfb-bc8493ae77d0-webhook-cert\") pod \"c65d990a-7650-476d-acfb-bc8493ae77d0\" (UID: \"c65d990a-7650-476d-acfb-bc8493ae77d0\") " Jul 22 20:19:42.665408 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.665279 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf4dt\" (UniqueName: \"kubernetes.io/projected/c65d990a-7650-476d-acfb-bc8493ae77d0-kube-api-access-rf4dt\") pod \"c65d990a-7650-476d-acfb-bc8493ae77d0\" (UID: \"c65d990a-7650-476d-acfb-bc8493ae77d0\") " Jul 22 20:19:42.667331 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.667300 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c65d990a-7650-476d-acfb-bc8493ae77d0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c65d990a-7650-476d-acfb-bc8493ae77d0" (UID: "c65d990a-7650-476d-acfb-bc8493ae77d0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 20:19:42.667444 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.667358 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65d990a-7650-476d-acfb-bc8493ae77d0-kube-api-access-rf4dt" (OuterVolumeSpecName: "kube-api-access-rf4dt") pod "c65d990a-7650-476d-acfb-bc8493ae77d0" (UID: "c65d990a-7650-476d-acfb-bc8493ae77d0"). InnerVolumeSpecName "kube-api-access-rf4dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:19:42.749880 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.749838 2575 generic.go:356] "Generic (PLEG): container finished" podID="c65d990a-7650-476d-acfb-bc8493ae77d0" containerID="6b04dc59f13b9075d6d791c522805e843f4af3074b17b062847864b37c847b21" exitCode=0 Jul 22 20:19:42.749880 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.749882 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64cc9776b5-kr86l" Jul 22 20:19:42.750104 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.749893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64cc9776b5-kr86l" event={"ID":"c65d990a-7650-476d-acfb-bc8493ae77d0","Type":"ContainerDied","Data":"6b04dc59f13b9075d6d791c522805e843f4af3074b17b062847864b37c847b21"} Jul 22 20:19:42.750104 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.749936 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64cc9776b5-kr86l" event={"ID":"c65d990a-7650-476d-acfb-bc8493ae77d0","Type":"ContainerDied","Data":"acbacbcd7847906aaf8bad6d763181eab45190e92451ab9818250bf84658c3af"} Jul 22 20:19:42.750104 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.749970 2575 scope.go:117] "RemoveContainer" containerID="6b04dc59f13b9075d6d791c522805e843f4af3074b17b062847864b37c847b21" Jul 22 20:19:42.758719 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.758702 2575 scope.go:117] "RemoveContainer" containerID="6b04dc59f13b9075d6d791c522805e843f4af3074b17b062847864b37c847b21" Jul 22 20:19:42.758970 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:19:42.758952 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b04dc59f13b9075d6d791c522805e843f4af3074b17b062847864b37c847b21\": container with ID starting with 6b04dc59f13b9075d6d791c522805e843f4af3074b17b062847864b37c847b21 not found: ID does not exist" containerID="6b04dc59f13b9075d6d791c522805e843f4af3074b17b062847864b37c847b21" Jul 22 20:19:42.759040 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.758999 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b04dc59f13b9075d6d791c522805e843f4af3074b17b062847864b37c847b21"} err="failed to get container status \"6b04dc59f13b9075d6d791c522805e843f4af3074b17b062847864b37c847b21\": rpc error: code = NotFound desc = could not find container \"6b04dc59f13b9075d6d791c522805e843f4af3074b17b062847864b37c847b21\": container with ID starting with 6b04dc59f13b9075d6d791c522805e843f4af3074b17b062847864b37c847b21 not found: ID does not exist" Jul 22 20:19:42.766670 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.766648 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c65d990a-7650-476d-acfb-bc8493ae77d0-webhook-cert\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:19:42.766670 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.766671 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf4dt\" (UniqueName: \"kubernetes.io/projected/c65d990a-7650-476d-acfb-bc8493ae77d0-kube-api-access-rf4dt\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:19:42.771039 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.771017 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64cc9776b5-kr86l"] Jul 22 20:19:42.778670 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:42.778650 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64cc9776b5-kr86l"] Jul 22 20:19:43.484964 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.484937 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:19:43.593492 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.593458 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64cc9776b5-sqx22"] Jul 22 20:19:43.593737 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.593725 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c65d990a-7650-476d-acfb-bc8493ae77d0" containerName="manager" Jul 22 20:19:43.593776 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.593739 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c65d990a-7650-476d-acfb-bc8493ae77d0" containerName="manager" Jul 22 20:19:43.593809 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.593794 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c65d990a-7650-476d-acfb-bc8493ae77d0" containerName="manager" Jul 22 20:19:43.597712 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.597691 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64cc9776b5-sqx22" Jul 22 20:19:43.601177 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.601146 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 20:19:43.601312 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.601188 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kkclf\"" Jul 22 20:19:43.601312 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.601189 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 20:19:43.606467 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.606445 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64cc9776b5-sqx22"] Jul 22 20:19:43.675971 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.675930 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd8hs\" (UniqueName: \"kubernetes.io/projected/e2d423d9-17cc-4c4a-b492-3416c04b332c-kube-api-access-dd8hs\") pod \"maas-controller-64cc9776b5-sqx22\" (UID: \"e2d423d9-17cc-4c4a-b492-3416c04b332c\") " pod="opendatahub/maas-controller-64cc9776b5-sqx22" Jul 22 20:19:43.676157 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.676070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2d423d9-17cc-4c4a-b492-3416c04b332c-webhook-cert\") pod \"maas-controller-64cc9776b5-sqx22\" (UID: \"e2d423d9-17cc-4c4a-b492-3416c04b332c\") " pod="opendatahub/maas-controller-64cc9776b5-sqx22" Jul 22 20:19:43.777439 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.777346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2d423d9-17cc-4c4a-b492-3416c04b332c-webhook-cert\") pod \"maas-controller-64cc9776b5-sqx22\" (UID: \"e2d423d9-17cc-4c4a-b492-3416c04b332c\") " pod="opendatahub/maas-controller-64cc9776b5-sqx22" Jul 22 20:19:43.777439 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.777394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd8hs\" (UniqueName: \"kubernetes.io/projected/e2d423d9-17cc-4c4a-b492-3416c04b332c-kube-api-access-dd8hs\") pod \"maas-controller-64cc9776b5-sqx22\" (UID: \"e2d423d9-17cc-4c4a-b492-3416c04b332c\") " pod="opendatahub/maas-controller-64cc9776b5-sqx22" Jul 22 20:19:43.779882 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.779855 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2d423d9-17cc-4c4a-b492-3416c04b332c-webhook-cert\") pod \"maas-controller-64cc9776b5-sqx22\" (UID: \"e2d423d9-17cc-4c4a-b492-3416c04b332c\") " pod="opendatahub/maas-controller-64cc9776b5-sqx22" Jul 22 20:19:43.786849 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.786828 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd8hs\" (UniqueName: \"kubernetes.io/projected/e2d423d9-17cc-4c4a-b492-3416c04b332c-kube-api-access-dd8hs\") pod \"maas-controller-64cc9776b5-sqx22\" (UID: \"e2d423d9-17cc-4c4a-b492-3416c04b332c\") " pod="opendatahub/maas-controller-64cc9776b5-sqx22" Jul 22 20:19:43.908016 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:43.907961 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64cc9776b5-sqx22" Jul 22 20:19:44.007133 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:44.007101 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c65d990a-7650-476d-acfb-bc8493ae77d0" path="/var/lib/kubelet/pods/c65d990a-7650-476d-acfb-bc8493ae77d0/volumes" Jul 22 20:19:44.050498 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:44.050476 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64cc9776b5-sqx22"] Jul 22 20:19:44.052810 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:19:44.052783 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d423d9_17cc_4c4a_b492_3416c04b332c.slice/crio-e58415a810aaf95449d1134f5609847e010176e02f45583ee30d628e2f7be379 WatchSource:0}: Error finding container e58415a810aaf95449d1134f5609847e010176e02f45583ee30d628e2f7be379: Status 404 returned error can't find the container with id e58415a810aaf95449d1134f5609847e010176e02f45583ee30d628e2f7be379 Jul 22 20:19:44.760267 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:44.760229 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64cc9776b5-sqx22" event={"ID":"e2d423d9-17cc-4c4a-b492-3416c04b332c","Type":"ContainerStarted","Data":"3b995561cae12af797e7d84ca77d3b27dba985c44b48cd79cf859710bd4b45ae"} Jul 22 20:19:44.760643 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:44.760285 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64cc9776b5-sqx22" event={"ID":"e2d423d9-17cc-4c4a-b492-3416c04b332c","Type":"ContainerStarted","Data":"e58415a810aaf95449d1134f5609847e010176e02f45583ee30d628e2f7be379"} Jul 22 20:19:44.760643 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:44.760343 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64cc9776b5-sqx22" Jul 22 20:19:44.781340 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:44.781285 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-64cc9776b5-sqx22" podStartSLOduration=1.464086609 podStartE2EDuration="1.781268006s" totalImagesPullingTime="317.181397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:19:44.779446184 +0000 UTC m=+1085.292325904" watchObservedRunningTime="2026-07-22 20:19:44.781268006 +0000 UTC m=+1085.294147735" Jul 22 20:19:55.767019 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:19:55.766964 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64cc9776b5-sqx22" Jul 22 20:20:05.883868 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:05.883830 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:20:18.685255 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:18.685218 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:20:19.189182 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:19.189139 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:20:19.293354 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:19.293322 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:20:19.397191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:19.397157 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:20:25.368737 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:25.368698 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:20:25.487551 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:25.487518 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:20:25.598146 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:25.598112 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:20:25.997666 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:25.997574 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:20:31.912616 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:31.912582 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:20:32.188382 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:32.188348 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:20:32.216612 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:32.216578 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:20:48.983292 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:48.983257 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:20:58.574109 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:20:58.574070 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:21:08.779083 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:21:08.779046 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:21:14.685646 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:21:14.685605 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:21:26.083617 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:21:26.083577 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:21:36.089468 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:21:36.089435 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:21:40.757785 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:21:40.757684 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:21:40.760931 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:21:40.759572 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:21:48.980797 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:21:48.980759 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:21:59.476319 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:21:59.476283 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:22:20.677951 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:22:20.677914 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:22:30.880327 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:22:30.880290 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:22:55.479686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:22:55.479608 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:23:08.616240 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:23:08.616206 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:23:10.132966 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:23:10.132932 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:23:11.590993 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:23:11.590944 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:23:24.082088 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:23:24.082051 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:23:33.590022 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:23:33.589990 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:23:43.282013 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:23:43.281967 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:23:46.225685 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:23:46.225649 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:23:53.084093 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:23:53.084048 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:24:02.681017 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:24:02.680915 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:25:49.676742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:25:49.676655 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:25:59.187187 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:25:59.187145 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:26:12.178914 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:26:12.178874 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:26:26.049722 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:26:26.049683 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:26:36.093683 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:26:36.093653 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:26:40.778615 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:26:40.778509 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:26:40.782753 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:26:40.779922 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:26:46.275218 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:26:46.275183 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:26:47.121309 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:26:47.121275 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:26:54.683225 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:26:54.683187 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:27:04.674882 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:27:04.674799 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:27:15.677155 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:27:15.677116 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:27:25.784843 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:27:25.784808 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:27:39.277865 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:27:39.277826 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:27:57.478634 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:27:57.478598 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:28:06.883213 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:06.883174 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:28:28.133610 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.133566 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv"] Jul 22 20:28:28.136700 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.136682 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.140007 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.139969 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tvrjw\"" Jul 22 20:28:28.140007 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.140004 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 20:28:28.140215 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.139968 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 20:28:28.140215 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.140052 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-55acace8-kserve-self-signed-certs\"" Jul 22 20:28:28.155167 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.155138 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv"] Jul 22 20:28:28.182895 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.182856 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:28:28.271094 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.271050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-home\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.271094 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.271094 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-dshm\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.271335 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.271191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nms8v\" (UniqueName: \"kubernetes.io/projected/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-kube-api-access-nms8v\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.271335 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.271217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-tmp-dir\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.271335 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.271244 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-model-cache\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.271447 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.271334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-tls-certs\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.372338 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.372302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nms8v\" (UniqueName: \"kubernetes.io/projected/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-kube-api-access-nms8v\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.372515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.372344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-tmp-dir\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.372515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.372370 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-model-cache\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.372515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.372405 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-tls-certs\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.372515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.372433 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-home\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.372515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.372449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-dshm\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.372789 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.372755 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-tmp-dir\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.372905 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.372881 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-model-cache\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.372963 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.372888 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-home\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.374879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.374857 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-dshm\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.375031 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.375015 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-tls-certs\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.379593 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.379570 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nms8v\" (UniqueName: \"kubernetes.io/projected/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-kube-api-access-nms8v\") pod \"e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.446251 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.446203 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:28.582287 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.582254 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv"] Jul 22 20:28:28.586848 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:28:28.586823 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bee0c9a_ce31_4f50_ab3c_8770aca1852b.slice/crio-63ce8a769b09892c23996bc9cbfae0eb2e3889ce1640ce93850d111a00b3b995 WatchSource:0}: Error finding container 63ce8a769b09892c23996bc9cbfae0eb2e3889ce1640ce93850d111a00b3b995: Status 404 returned error can't find the container with id 63ce8a769b09892c23996bc9cbfae0eb2e3889ce1640ce93850d111a00b3b995 Jul 22 20:28:28.589161 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.589145 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 20:28:28.803446 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:28.803364 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" event={"ID":"4bee0c9a-ce31-4f50-ab3c-8770aca1852b","Type":"ContainerStarted","Data":"63ce8a769b09892c23996bc9cbfae0eb2e3889ce1640ce93850d111a00b3b995"} Jul 22 20:28:29.809701 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:29.809637 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" event={"ID":"4bee0c9a-ce31-4f50-ab3c-8770aca1852b","Type":"ContainerStarted","Data":"1bc0953b1f9af9e16080517bd45d06f76dd5bad6f730645fcbc950895a9b1a91"} Jul 22 20:28:29.826997 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:29.826925 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" podStartSLOduration=0.726074535 podStartE2EDuration="1.826906563s" totalImagesPullingTime="1.100832028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:28:29.825318969 +0000 UTC m=+1610.338198676" watchObservedRunningTime="2026-07-22 20:28:29.826906563 +0000 UTC m=+1610.339786268" Jul 22 20:28:32.378053 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:32.377957 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:28:32.425923 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:32.425890 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:28:38.447159 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:38.447115 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:38.447517 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:38.447177 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:38.464406 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:38.464372 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:38.859001 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:38.858907 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:28:43.637779 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:43.637743 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:28:47.573896 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:47.573853 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:28:52.535573 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:52.535535 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv"] Jul 22 20:28:52.578373 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:52.578341 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:28:54.336756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:54.336721 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:28:58.068107 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:28:58.068075 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:29:07.555936 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.555888 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" podUID="4bee0c9a-ce31-4f50-ab3c-8770aca1852b" containerName="main" containerID="cri-o://1bc0953b1f9af9e16080517bd45d06f76dd5bad6f730645fcbc950895a9b1a91" gracePeriod=45 Jul 22 20:29:07.789791 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.789767 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:29:07.909742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.909657 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-home\") pod \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " Jul 22 20:29:07.909742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.909700 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-model-cache\") pod \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " Jul 22 20:29:07.909742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.909730 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-tmp-dir\") pod \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " Jul 22 20:29:07.909742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.909747 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-dshm\") pod \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " Jul 22 20:29:07.910075 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.909800 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-tls-certs\") pod \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " Jul 22 20:29:07.910075 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.909826 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nms8v\" (UniqueName: \"kubernetes.io/projected/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-kube-api-access-nms8v\") pod \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\" (UID: \"4bee0c9a-ce31-4f50-ab3c-8770aca1852b\") " Jul 22 20:29:07.910075 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.909965 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-home" (OuterVolumeSpecName: "home") pod "4bee0c9a-ce31-4f50-ab3c-8770aca1852b" (UID: "4bee0c9a-ce31-4f50-ab3c-8770aca1852b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:29:07.910075 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.910003 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-model-cache" (OuterVolumeSpecName: "model-cache") pod "4bee0c9a-ce31-4f50-ab3c-8770aca1852b" (UID: "4bee0c9a-ce31-4f50-ab3c-8770aca1852b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:29:07.910075 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.910064 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-home\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:29:07.910303 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.910072 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4bee0c9a-ce31-4f50-ab3c-8770aca1852b" (UID: "4bee0c9a-ce31-4f50-ab3c-8770aca1852b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:29:07.912139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.912114 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-dshm" (OuterVolumeSpecName: "dshm") pod "4bee0c9a-ce31-4f50-ab3c-8770aca1852b" (UID: "4bee0c9a-ce31-4f50-ab3c-8770aca1852b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:29:07.912221 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.912153 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-kube-api-access-nms8v" (OuterVolumeSpecName: "kube-api-access-nms8v") pod "4bee0c9a-ce31-4f50-ab3c-8770aca1852b" (UID: "4bee0c9a-ce31-4f50-ab3c-8770aca1852b"). InnerVolumeSpecName "kube-api-access-nms8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:29:07.912221 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.912171 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4bee0c9a-ce31-4f50-ab3c-8770aca1852b" (UID: "4bee0c9a-ce31-4f50-ab3c-8770aca1852b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 20:29:07.962298 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.962263 2575 generic.go:356] "Generic (PLEG): container finished" podID="4bee0c9a-ce31-4f50-ab3c-8770aca1852b" containerID="1bc0953b1f9af9e16080517bd45d06f76dd5bad6f730645fcbc950895a9b1a91" exitCode=0 Jul 22 20:29:07.962434 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.962312 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" Jul 22 20:29:07.962434 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.962352 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" event={"ID":"4bee0c9a-ce31-4f50-ab3c-8770aca1852b","Type":"ContainerDied","Data":"1bc0953b1f9af9e16080517bd45d06f76dd5bad6f730645fcbc950895a9b1a91"} Jul 22 20:29:07.962434 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.962407 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv" event={"ID":"4bee0c9a-ce31-4f50-ab3c-8770aca1852b","Type":"ContainerDied","Data":"63ce8a769b09892c23996bc9cbfae0eb2e3889ce1640ce93850d111a00b3b995"} Jul 22 20:29:07.962434 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.962430 2575 scope.go:117] "RemoveContainer" containerID="1bc0953b1f9af9e16080517bd45d06f76dd5bad6f730645fcbc950895a9b1a91" Jul 22 20:29:07.970847 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.970831 2575 scope.go:117] "RemoveContainer" containerID="1bc0953b1f9af9e16080517bd45d06f76dd5bad6f730645fcbc950895a9b1a91" Jul 22 20:29:07.971133 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:29:07.971114 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc0953b1f9af9e16080517bd45d06f76dd5bad6f730645fcbc950895a9b1a91\": container with ID starting with 1bc0953b1f9af9e16080517bd45d06f76dd5bad6f730645fcbc950895a9b1a91 not found: ID does not exist" containerID="1bc0953b1f9af9e16080517bd45d06f76dd5bad6f730645fcbc950895a9b1a91" Jul 22 20:29:07.971185 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.971141 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc0953b1f9af9e16080517bd45d06f76dd5bad6f730645fcbc950895a9b1a91"} err="failed to get container status \"1bc0953b1f9af9e16080517bd45d06f76dd5bad6f730645fcbc950895a9b1a91\": rpc error: code = NotFound desc = could not find container \"1bc0953b1f9af9e16080517bd45d06f76dd5bad6f730645fcbc950895a9b1a91\": container with ID starting with 1bc0953b1f9af9e16080517bd45d06f76dd5bad6f730645fcbc950895a9b1a91 not found: ID does not exist" Jul 22 20:29:07.982381 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.982362 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv"] Jul 22 20:29:07.985992 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:07.985956 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-55acace8-kserve-bfcd99d4c-28kbv"] Jul 22 20:29:08.006696 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:08.006676 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bee0c9a-ce31-4f50-ab3c-8770aca1852b" path="/var/lib/kubelet/pods/4bee0c9a-ce31-4f50-ab3c-8770aca1852b/volumes" Jul 22 20:29:08.010631 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:08.010616 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-model-cache\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:29:08.010631 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:08.010633 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-tmp-dir\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:29:08.010734 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:08.010641 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-dshm\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:29:08.010734 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:08.010648 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-tls-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:29:08.010734 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:08.010656 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nms8v\" (UniqueName: \"kubernetes.io/projected/4bee0c9a-ce31-4f50-ab3c-8770aca1852b-kube-api-access-nms8v\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:29:11.588662 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:11.588627 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:29:17.991883 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:17.991852 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:29:34.470409 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:34.470372 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:29:44.079117 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:44.079085 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:29:53.670352 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:29:53.670319 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:30:00.135680 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:00.135644 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745870-m4zxk"] Jul 22 20:30:00.136194 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:00.135935 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bee0c9a-ce31-4f50-ab3c-8770aca1852b" containerName="main" Jul 22 20:30:00.136194 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:00.135946 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bee0c9a-ce31-4f50-ab3c-8770aca1852b" containerName="main" Jul 22 20:30:00.136194 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:00.136018 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bee0c9a-ce31-4f50-ab3c-8770aca1852b" containerName="main" Jul 22 20:30:00.138763 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:00.138747 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745870-m4zxk" Jul 22 20:30:00.140778 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:00.140758 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjqt7\"" Jul 22 20:30:00.145754 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:00.145700 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745870-m4zxk"] Jul 22 20:30:00.148924 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:00.148906 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745870-m4zxk" Jul 22 20:30:00.281405 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:00.281382 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745870-m4zxk"] Jul 22 20:30:00.283808 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:30:00.283782 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0e23b8_3ec2_43a8_937c_ee4da5f6a63b.slice/crio-d384faf33f63e32af2effa2d1c4d1a02656bc5f842f65e0f7d780372e385be58 WatchSource:0}: Error finding container d384faf33f63e32af2effa2d1c4d1a02656bc5f842f65e0f7d780372e385be58: Status 404 returned error can't find the container with id d384faf33f63e32af2effa2d1c4d1a02656bc5f842f65e0f7d780372e385be58 Jul 22 20:30:01.166214 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:01.166182 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745870-m4zxk" event={"ID":"3b0e23b8-3ec2-43a8-937c-ee4da5f6a63b","Type":"ContainerStarted","Data":"d384faf33f63e32af2effa2d1c4d1a02656bc5f842f65e0f7d780372e385be58"} Jul 22 20:30:03.175901 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:03.175806 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745870-m4zxk" event={"ID":"3b0e23b8-3ec2-43a8-937c-ee4da5f6a63b","Type":"ContainerStarted","Data":"cfbbdcd53bbcbc8c82aeeb10894f04634751d54c7e67e89fbbd2fb41bf6edc66"} Jul 22 20:30:03.194567 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:03.194520 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745870-m4zxk" podStartSLOduration=0.911746216 podStartE2EDuration="3.194503781s" totalImagesPullingTime="2.282757565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:30:03.19365175 +0000 UTC m=+1703.706531456" watchObservedRunningTime="2026-07-22 20:30:03.194503781 +0000 UTC m=+1703.707383464" Jul 22 20:30:04.186129 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:04.186093 2575 generic.go:356] "Generic (PLEG): container finished" podID="3b0e23b8-3ec2-43a8-937c-ee4da5f6a63b" containerID="cfbbdcd53bbcbc8c82aeeb10894f04634751d54c7e67e89fbbd2fb41bf6edc66" exitCode=0 Jul 22 20:30:04.186565 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:04.186167 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745870-m4zxk" event={"ID":"3b0e23b8-3ec2-43a8-937c-ee4da5f6a63b","Type":"ContainerDied","Data":"cfbbdcd53bbcbc8c82aeeb10894f04634751d54c7e67e89fbbd2fb41bf6edc66"} Jul 22 20:30:04.987751 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:04.987715 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:30:05.314659 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:05.314636 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745870-m4zxk" Jul 22 20:30:06.195270 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:06.195238 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745870-m4zxk" event={"ID":"3b0e23b8-3ec2-43a8-937c-ee4da5f6a63b","Type":"ContainerDied","Data":"d384faf33f63e32af2effa2d1c4d1a02656bc5f842f65e0f7d780372e385be58"} Jul 22 20:30:06.195270 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:06.195257 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745870-m4zxk" Jul 22 20:30:06.195463 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:06.195286 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d384faf33f63e32af2effa2d1c4d1a02656bc5f842f65e0f7d780372e385be58" Jul 22 20:30:14.381043 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:14.381010 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:30:24.476961 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:24.476921 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:30:36.776669 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:36.776633 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:30:46.288537 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:46.288502 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:30:56.379780 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:30:56.379745 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:31:05.996928 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:31:05.996888 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:31:17.175063 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:31:17.175022 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:31:27.578609 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:31:27.578572 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:31:27.632059 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:31:27.632021 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:31:27.720434 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:31:27.720395 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:31:39.668953 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:31:39.668922 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:31:40.802895 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:31:40.802776 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:31:40.810572 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:31:40.804615 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:31:51.975954 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:31:51.975916 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:32:04.072787 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:04.072751 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:32:06.025924 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:06.025891 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:18.276354 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:18.276315 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:32:27.458686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:27.458653 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:30.486039 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:30.486005 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:30.583203 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:30.583166 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:31.105091 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:31.105054 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:31.379753 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:31.379670 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:32:36.807223 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:36.807180 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:36.909275 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:36.909243 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:37.070804 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.070722 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:37.333521 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.333450 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29"] Jul 22 20:32:37.333747 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.333734 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0e23b8-3ec2-43a8-937c-ee4da5f6a63b" containerName="cleanup" Jul 22 20:32:37.333788 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.333749 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0e23b8-3ec2-43a8-937c-ee4da5f6a63b" containerName="cleanup" Jul 22 20:32:37.333824 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.333797 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b0e23b8-3ec2-43a8-937c-ee4da5f6a63b" containerName="cleanup" Jul 22 20:32:37.336685 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.336670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.338888 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.338869 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e8c31dad-openshift-default-dockercfg-5z6q9\"" Jul 22 20:32:37.360113 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.360085 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29"] Jul 22 20:32:37.363539 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.363514 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:37.484579 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.484548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-workload-socket\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.484738 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.484587 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-credential-socket\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.484738 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.484618 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-token\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.484738 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.484712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-envoy\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.484871 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.484744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-workload-certs\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.484871 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.484769 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istiod-ca-cert\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.484871 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.484787 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-podinfo\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.484871 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.484841 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-data\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.485021 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.484886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98xn5\" (UniqueName: \"kubernetes.io/projected/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-kube-api-access-98xn5\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.585602 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.585509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istiod-ca-cert\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.585602 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.585559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-podinfo\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.585602 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.585588 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-data\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.585868 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.585612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xn5\" (UniqueName: \"kubernetes.io/projected/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-kube-api-access-98xn5\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.585868 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.585639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-workload-socket\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.585868 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.585671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-credential-socket\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.585868 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.585707 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-token\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.585868 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.585746 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-envoy\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.585868 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.585773 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-workload-certs\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.586195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.586037 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-data\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.586195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.586088 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-workload-socket\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.586195 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.586149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-workload-certs\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.586457 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.586433 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-credential-socket\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.586532 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.586438 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istiod-ca-cert\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.587654 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.587626 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:37.588368 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.588345 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-envoy\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.588477 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.588459 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-podinfo\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.593501 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.593477 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-token\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.593653 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.593633 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98xn5\" (UniqueName: \"kubernetes.io/projected/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-kube-api-access-98xn5\") pod \"e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.646671 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.646645 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:37.774851 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.774822 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29"] Jul 22 20:32:37.778135 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:32:37.778107 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4fffe3_ad06_4ae1_84cf_2aa0461a37be.slice/crio-9de374296f2ac08619d66f654248870306500aa8f73447342aa0b333e4d5f08e WatchSource:0}: Error finding container 9de374296f2ac08619d66f654248870306500aa8f73447342aa0b333e4d5f08e: Status 404 returned error can't find the container with id 9de374296f2ac08619d66f654248870306500aa8f73447342aa0b333e4d5f08e Jul 22 20:32:37.780253 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.780211 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:32:37.780339 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.780295 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:32:37.780381 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.780334 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:32:37.797921 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:37.797899 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" event={"ID":"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be","Type":"ContainerStarted","Data":"9de374296f2ac08619d66f654248870306500aa8f73447342aa0b333e4d5f08e"} Jul 22 20:32:38.802607 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:38.802574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" event={"ID":"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be","Type":"ContainerStarted","Data":"6c3bb4e5e742141a5f2b9afb9a530a8b1696ff1613025889bd26d5fdc3e282b3"} Jul 22 20:32:38.825752 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:38.825703 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" podStartSLOduration=1.825688985 podStartE2EDuration="1.825688985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:32:38.823559402 +0000 UTC m=+1859.336439107" watchObservedRunningTime="2026-07-22 20:32:38.825688985 +0000 UTC m=+1859.338568691" Jul 22 20:32:39.647370 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:39.647337 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:39.652043 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:39.652020 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:39.806425 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:39.806391 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:39.807289 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:39.807268 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:40.181803 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:40.181764 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:40.280668 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:40.280636 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:40.395043 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:40.395002 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:40.750839 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:40.750808 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:43.264960 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:43.264914 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:43.366720 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:43.366678 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:32:47.782086 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:47.782050 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:32:55.408237 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:55.408198 2575 patch_prober.go:28] interesting pod/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": read tcp 10.133.0.2:35648->10.133.0.22:15021: read: connection reset by peer" start-of-body= Jul 22 20:32:55.408603 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:55.408265 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" podUID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": read tcp 10.133.0.2:35648->10.133.0.22:15021: read: connection reset by peer" Jul 22 20:32:55.880281 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:55.880247 2575 generic.go:356] "Generic (PLEG): container finished" podID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerID="6c3bb4e5e742141a5f2b9afb9a530a8b1696ff1613025889bd26d5fdc3e282b3" exitCode=0 Jul 22 20:32:55.880456 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:55.880311 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" event={"ID":"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be","Type":"ContainerDied","Data":"6c3bb4e5e742141a5f2b9afb9a530a8b1696ff1613025889bd26d5fdc3e282b3"} Jul 22 20:32:55.880692 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:55.880679 2575 scope.go:117] "RemoveContainer" containerID="6c3bb4e5e742141a5f2b9afb9a530a8b1696ff1613025889bd26d5fdc3e282b3" Jul 22 20:32:55.881670 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:55.881641 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:32:55.881725 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:55.881712 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:32:55.881758 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:55.881746 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:32:56.886484 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:56.886452 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" event={"ID":"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be","Type":"ContainerStarted","Data":"af1679e24a7c0d294f101816f8e7e9b5b378b30b8b2046df998531f362bc7d73"} Jul 22 20:32:57.647919 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:57.647879 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:32:57.648183 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:57.648159 2575 patch_prober.go:28] interesting pod/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Jul 22 20:32:57.648232 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:57.648211 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" podUID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Jul 22 20:32:58.647190 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:58.647158 2575 patch_prober.go:28] interesting pod/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Jul 22 20:32:58.647554 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:58.647209 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" podUID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Jul 22 20:32:59.647485 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:59.647447 2575 patch_prober.go:28] interesting pod/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Jul 22 20:32:59.647870 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:32:59.647502 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" podUID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Jul 22 20:33:00.648146 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:00.648108 2575 patch_prober.go:28] interesting pod/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Jul 22 20:33:00.648535 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:00.648165 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" podUID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Jul 22 20:33:01.651607 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:01.651528 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:33:01.651953 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:01.651819 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:33:01.652564 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:01.652543 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:33:05.856383 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:05.856350 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x"] Jul 22 20:33:05.858433 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:05.858417 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x" Jul 22 20:33:05.860679 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:05.860655 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 20:33:05.860786 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:05.860667 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjqt7\"" Jul 22 20:33:05.866269 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:05.866235 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x"] Jul 22 20:33:05.926777 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:05.926746 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7df038-b56e-4866-b5f3-eb74c11ed482-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x\" (UID: \"3b7df038-b56e-4866-b5f3-eb74c11ed482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x" Jul 22 20:33:06.028108 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:06.028078 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7df038-b56e-4866-b5f3-eb74c11ed482-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x\" (UID: \"3b7df038-b56e-4866-b5f3-eb74c11ed482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x" Jul 22 20:33:06.028773 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:06.028750 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7df038-b56e-4866-b5f3-eb74c11ed482-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x\" (UID: \"3b7df038-b56e-4866-b5f3-eb74c11ed482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x" Jul 22 20:33:06.169351 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:06.169261 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x" Jul 22 20:33:06.294764 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:06.294616 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x"] Jul 22 20:33:06.297584 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:33:06.297552 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7df038_b56e_4866_b5f3_eb74c11ed482.slice/crio-fe8cbceaf85b1e1fd229ddcc48d802115fe22d5f483c837aaf03f66b432c4788 WatchSource:0}: Error finding container fe8cbceaf85b1e1fd229ddcc48d802115fe22d5f483c837aaf03f66b432c4788: Status 404 returned error can't find the container with id fe8cbceaf85b1e1fd229ddcc48d802115fe22d5f483c837aaf03f66b432c4788 Jul 22 20:33:06.934448 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:06.934371 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x" event={"ID":"3b7df038-b56e-4866-b5f3-eb74c11ed482","Type":"ContainerStarted","Data":"f0cec09b31cfad83147902e497c022ee197257c5f88a10c82fc27389d7c8cf66"} Jul 22 20:33:06.934795 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:06.934450 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x" event={"ID":"3b7df038-b56e-4866-b5f3-eb74c11ed482","Type":"ContainerStarted","Data":"fe8cbceaf85b1e1fd229ddcc48d802115fe22d5f483c837aaf03f66b432c4788"} Jul 22 20:33:06.949403 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:06.949358 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x" podStartSLOduration=1.9493436769999999 podStartE2EDuration="1.949343677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:33:06.948446343 +0000 UTC m=+1887.461326049" watchObservedRunningTime="2026-07-22 20:33:06.949343677 +0000 UTC m=+1887.462223382" Jul 22 20:33:07.939993 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:07.939937 2575 generic.go:356] "Generic (PLEG): container finished" podID="3b7df038-b56e-4866-b5f3-eb74c11ed482" containerID="f0cec09b31cfad83147902e497c022ee197257c5f88a10c82fc27389d7c8cf66" exitCode=0 Jul 22 20:33:07.940458 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:07.940016 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x" event={"ID":"3b7df038-b56e-4866-b5f3-eb74c11ed482","Type":"ContainerDied","Data":"f0cec09b31cfad83147902e497c022ee197257c5f88a10c82fc27389d7c8cf66"} Jul 22 20:33:09.067586 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:09.067562 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x" Jul 22 20:33:09.156750 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:09.156713 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7df038-b56e-4866-b5f3-eb74c11ed482-maas-api-service-ca\") pod \"3b7df038-b56e-4866-b5f3-eb74c11ed482\" (UID: \"3b7df038-b56e-4866-b5f3-eb74c11ed482\") " Jul 22 20:33:09.157086 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:09.157065 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7df038-b56e-4866-b5f3-eb74c11ed482-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b7df038-b56e-4866-b5f3-eb74c11ed482" (UID: "3b7df038-b56e-4866-b5f3-eb74c11ed482"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:33:09.257347 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:09.257260 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7df038-b56e-4866-b5f3-eb74c11ed482-maas-api-service-ca\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:33:09.949393 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:09.949365 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x" Jul 22 20:33:09.949557 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:09.949396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x" event={"ID":"3b7df038-b56e-4866-b5f3-eb74c11ed482","Type":"ContainerDied","Data":"fe8cbceaf85b1e1fd229ddcc48d802115fe22d5f483c837aaf03f66b432c4788"} Jul 22 20:33:09.949557 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:09.949438 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe8cbceaf85b1e1fd229ddcc48d802115fe22d5f483c837aaf03f66b432c4788" Jul 22 20:33:16.372521 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:16.372488 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:19.172350 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:19.172308 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:33:20.921872 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:20.921837 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:20.929052 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:20.929025 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x"] Jul 22 20:33:20.931544 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:20.931520 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8c31dad-fkj2x"] Jul 22 20:33:21.062067 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:21.062037 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29"] Jul 22 20:33:21.062253 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:21.062227 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" podUID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerName="istio-proxy" containerID="cri-o://af1679e24a7c0d294f101816f8e7e9b5b378b30b8b2046df998531f362bc7d73" gracePeriod=30 Jul 22 20:33:21.085625 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:21.085579 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:21.315047 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:21.314972 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:21.578383 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:21.578282 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:21.681285 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:21.681253 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:21.794254 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:21.794221 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:22.006124 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.006084 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b7df038-b56e-4866-b5f3-eb74c11ed482" path="/var/lib/kubelet/pods/3b7df038-b56e-4866-b5f3-eb74c11ed482/volumes" Jul 22 20:33:22.074332 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.074303 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75"] Jul 22 20:33:22.074643 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.074624 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b7df038-b56e-4866-b5f3-eb74c11ed482" containerName="revoke-keys" Jul 22 20:33:22.074643 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.074640 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b7df038-b56e-4866-b5f3-eb74c11ed482" containerName="revoke-keys" Jul 22 20:33:22.074838 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.074689 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b7df038-b56e-4866-b5f3-eb74c11ed482" containerName="revoke-keys" Jul 22 20:33:22.076953 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.076933 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.079104 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.079087 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cef58190-gw-openshift-default-dockercfg-z5mnc\"" Jul 22 20:33:22.089870 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.089843 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75"] Jul 22 20:33:22.110734 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.110692 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:22.166863 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.166826 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a6e3d9-77c5-432b-8487-8ca455700237-istio-token\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.167059 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.166871 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-workload-certs\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.167059 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.166890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-istio-envoy\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.167059 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.166935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-istio-data\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.167059 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.166989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-workload-socket\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.167059 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.167036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a6e3d9-77c5-432b-8487-8ca455700237-istiod-ca-cert\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.167227 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.167094 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a6e3d9-77c5-432b-8487-8ca455700237-istio-podinfo\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.167227 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.167138 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-credential-socket\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.167227 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.167164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xqr\" (UniqueName: \"kubernetes.io/projected/88a6e3d9-77c5-432b-8487-8ca455700237-kube-api-access-t7xqr\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.229787 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.229732 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:22.268578 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.268480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a6e3d9-77c5-432b-8487-8ca455700237-istiod-ca-cert\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.268578 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.268533 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a6e3d9-77c5-432b-8487-8ca455700237-istio-podinfo\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.268578 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.268564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-credential-socket\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.268842 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.268583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xqr\" (UniqueName: \"kubernetes.io/projected/88a6e3d9-77c5-432b-8487-8ca455700237-kube-api-access-t7xqr\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.268842 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.268627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a6e3d9-77c5-432b-8487-8ca455700237-istio-token\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.268842 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.268653 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-workload-certs\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.268842 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.268671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-istio-envoy\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.268842 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.268784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-istio-data\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.268842 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.268837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-workload-socket\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.269179 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.269083 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-credential-socket\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.269241 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.269205 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-istio-data\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.269295 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.269262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-workload-certs\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.269351 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.269292 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-workload-socket\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.269400 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.269348 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a6e3d9-77c5-432b-8487-8ca455700237-istiod-ca-cert\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.270907 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.270887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-istio-envoy\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.271254 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.271234 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a6e3d9-77c5-432b-8487-8ca455700237-istio-podinfo\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.280827 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.280787 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a6e3d9-77c5-432b-8487-8ca455700237-istio-token\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.281104 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.281082 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xqr\" (UniqueName: \"kubernetes.io/projected/88a6e3d9-77c5-432b-8487-8ca455700237-kube-api-access-t7xqr\") pod \"e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.387879 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.387834 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:22.534192 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.534161 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75"] Jul 22 20:33:22.535876 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:33:22.535848 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a6e3d9_77c5_432b_8487_8ca455700237.slice/crio-ad5359ac111c96a8e57034b7fe7df50de31d64e04f9add18bf0b3e54d88ecb76 WatchSource:0}: Error finding container ad5359ac111c96a8e57034b7fe7df50de31d64e04f9add18bf0b3e54d88ecb76: Status 404 returned error can't find the container with id ad5359ac111c96a8e57034b7fe7df50de31d64e04f9add18bf0b3e54d88ecb76 Jul 22 20:33:22.537851 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.537817 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:33:22.537938 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.537898 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:33:22.538029 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:22.537948 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:33:23.007585 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:23.007548 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" event={"ID":"88a6e3d9-77c5-432b-8487-8ca455700237","Type":"ContainerStarted","Data":"5d879f9c94f124b302057e56515313c73dc5956502ce96fc3227443ddfe7d494"} Jul 22 20:33:23.008079 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:23.007616 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" event={"ID":"88a6e3d9-77c5-432b-8487-8ca455700237","Type":"ContainerStarted","Data":"ad5359ac111c96a8e57034b7fe7df50de31d64e04f9add18bf0b3e54d88ecb76"} Jul 22 20:33:23.031488 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:23.031440 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" podStartSLOduration=1.03142462 podStartE2EDuration="1.03142462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:33:23.029858478 +0000 UTC m=+1903.542738187" watchObservedRunningTime="2026-07-22 20:33:23.03142462 +0000 UTC m=+1903.544304326" Jul 22 20:33:23.389035 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:23.388929 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:23.390500 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:23.390472 2575 patch_prober.go:28] interesting pod/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 22 20:33:23.390634 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:23.390530 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" podUID="88a6e3d9-77c5-432b-8487-8ca455700237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 22 20:33:24.392653 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:24.392624 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:24.504754 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:24.504705 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:24.624655 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:24.624612 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:25.016046 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:25.016013 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:25.022742 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:25.022713 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:25.095771 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:25.095734 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:25.203756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:25.203720 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:25.295833 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:25.295750 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:33:26.310268 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.310239 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:33:26.404167 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404051 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-workload-socket\") pod \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " Jul 22 20:33:26.404167 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404114 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-envoy\") pod \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " Jul 22 20:33:26.404167 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404150 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-podinfo\") pod \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " Jul 22 20:33:26.404447 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404175 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-credential-socket\") pod \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " Jul 22 20:33:26.404447 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404208 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-token\") pod \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " Jul 22 20:33:26.404447 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404230 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-data\") pod \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " Jul 22 20:33:26.404447 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404293 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istiod-ca-cert\") pod \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " Jul 22 20:33:26.404447 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404332 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98xn5\" (UniqueName: \"kubernetes.io/projected/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-kube-api-access-98xn5\") pod \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " Jul 22 20:33:26.404447 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404392 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-workload-certs\") pod \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\" (UID: \"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be\") " Jul 22 20:33:26.404718 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404479 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" (UID: "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:33:26.404718 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404664 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" (UID: "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:33:26.404718 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404677 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-credential-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:33:26.404871 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404780 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" (UID: "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:33:26.404926 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.404897 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" (UID: "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:33:26.405206 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.405180 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" (UID: "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:33:26.406649 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.406621 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-kube-api-access-98xn5" (OuterVolumeSpecName: "kube-api-access-98xn5") pod "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" (UID: "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be"). InnerVolumeSpecName "kube-api-access-98xn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:33:26.406737 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.406646 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" (UID: "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:33:26.406825 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.406809 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" (UID: "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 20:33:26.406864 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.406829 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" (UID: "ec4fffe3-ad06-4ae1-84cf-2aa0461a37be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:33:26.505442 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.505401 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-workload-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:33:26.505442 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.505441 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-envoy\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:33:26.505644 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.505454 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-podinfo\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:33:26.505644 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.505461 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-token\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:33:26.505644 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.505470 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istio-data\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:33:26.505644 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.505478 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-istiod-ca-cert\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:33:26.505644 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.505486 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98xn5\" (UniqueName: \"kubernetes.io/projected/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-kube-api-access-98xn5\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:33:26.505644 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:26.505495 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be-workload-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:33:27.025277 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:27.025244 2575 generic.go:356] "Generic (PLEG): container finished" podID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerID="af1679e24a7c0d294f101816f8e7e9b5b378b30b8b2046df998531f362bc7d73" exitCode=0 Jul 22 20:33:27.025476 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:27.025293 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" Jul 22 20:33:27.025476 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:27.025330 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" event={"ID":"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be","Type":"ContainerDied","Data":"af1679e24a7c0d294f101816f8e7e9b5b378b30b8b2046df998531f362bc7d73"} Jul 22 20:33:27.025476 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:27.025394 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29" event={"ID":"ec4fffe3-ad06-4ae1-84cf-2aa0461a37be","Type":"ContainerDied","Data":"9de374296f2ac08619d66f654248870306500aa8f73447342aa0b333e4d5f08e"} Jul 22 20:33:27.025476 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:27.025424 2575 scope.go:117] "RemoveContainer" containerID="af1679e24a7c0d294f101816f8e7e9b5b378b30b8b2046df998531f362bc7d73" Jul 22 20:33:27.034407 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:27.034388 2575 scope.go:117] "RemoveContainer" containerID="6c3bb4e5e742141a5f2b9afb9a530a8b1696ff1613025889bd26d5fdc3e282b3" Jul 22 20:33:27.041863 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:27.041844 2575 scope.go:117] "RemoveContainer" containerID="af1679e24a7c0d294f101816f8e7e9b5b378b30b8b2046df998531f362bc7d73" Jul 22 20:33:27.042157 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:33:27.042134 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1679e24a7c0d294f101816f8e7e9b5b378b30b8b2046df998531f362bc7d73\": container with ID starting with af1679e24a7c0d294f101816f8e7e9b5b378b30b8b2046df998531f362bc7d73 not found: ID does not exist" containerID="af1679e24a7c0d294f101816f8e7e9b5b378b30b8b2046df998531f362bc7d73" Jul 22 20:33:27.042248 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:27.042165 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1679e24a7c0d294f101816f8e7e9b5b378b30b8b2046df998531f362bc7d73"} err="failed to get container status \"af1679e24a7c0d294f101816f8e7e9b5b378b30b8b2046df998531f362bc7d73\": rpc error: code = NotFound desc = could not find container \"af1679e24a7c0d294f101816f8e7e9b5b378b30b8b2046df998531f362bc7d73\": container with ID starting with af1679e24a7c0d294f101816f8e7e9b5b378b30b8b2046df998531f362bc7d73 not found: ID does not exist" Jul 22 20:33:27.042248 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:27.042182 2575 scope.go:117] "RemoveContainer" containerID="6c3bb4e5e742141a5f2b9afb9a530a8b1696ff1613025889bd26d5fdc3e282b3" Jul 22 20:33:27.042452 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:33:27.042435 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3bb4e5e742141a5f2b9afb9a530a8b1696ff1613025889bd26d5fdc3e282b3\": container with ID starting with 6c3bb4e5e742141a5f2b9afb9a530a8b1696ff1613025889bd26d5fdc3e282b3 not found: ID does not exist" containerID="6c3bb4e5e742141a5f2b9afb9a530a8b1696ff1613025889bd26d5fdc3e282b3" Jul 22 20:33:27.042499 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:27.042459 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3bb4e5e742141a5f2b9afb9a530a8b1696ff1613025889bd26d5fdc3e282b3"} err="failed to get container status \"6c3bb4e5e742141a5f2b9afb9a530a8b1696ff1613025889bd26d5fdc3e282b3\": rpc error: code = NotFound desc = could not find container \"6c3bb4e5e742141a5f2b9afb9a530a8b1696ff1613025889bd26d5fdc3e282b3\": container with ID starting with 6c3bb4e5e742141a5f2b9afb9a530a8b1696ff1613025889bd26d5fdc3e282b3 not found: ID does not exist" Jul 22 20:33:27.045776 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:27.045756 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29"] Jul 22 20:33:27.049959 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:27.049941 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e8c31dad-openshift-default-7b675757c7-plt29"] Jul 22 20:33:28.006317 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:28.006287 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" path="/var/lib/kubelet/pods/ec4fffe3-ad06-4ae1-84cf-2aa0461a37be/volumes" Jul 22 20:33:30.178560 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:30.178526 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:33:38.072151 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:38.072069 2575 generic.go:356] "Generic (PLEG): container finished" podID="88a6e3d9-77c5-432b-8487-8ca455700237" containerID="5d879f9c94f124b302057e56515313c73dc5956502ce96fc3227443ddfe7d494" exitCode=0 Jul 22 20:33:38.072151 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:38.072123 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" event={"ID":"88a6e3d9-77c5-432b-8487-8ca455700237","Type":"ContainerDied","Data":"5d879f9c94f124b302057e56515313c73dc5956502ce96fc3227443ddfe7d494"} Jul 22 20:33:38.072576 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:38.072509 2575 scope.go:117] "RemoveContainer" containerID="5d879f9c94f124b302057e56515313c73dc5956502ce96fc3227443ddfe7d494" Jul 22 20:33:38.073467 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:38.073439 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:33:38.073520 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:38.073507 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:33:38.073568 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:38.073549 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:33:39.078438 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:39.078404 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" event={"ID":"88a6e3d9-77c5-432b-8487-8ca455700237","Type":"ContainerStarted","Data":"e175579698098c56af521fb3713426368fb994741c45005531d077295b680786"} Jul 22 20:33:39.388623 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:39.388527 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:39.389720 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:39.389694 2575 patch_prober.go:28] interesting pod/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 22 20:33:39.389838 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:39.389751 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" podUID="88a6e3d9-77c5-432b-8487-8ca455700237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 22 20:33:40.016711 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:40.016678 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:40.388652 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:40.388556 2575 patch_prober.go:28] interesting pod/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 22 20:33:40.388652 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:40.388620 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" podUID="88a6e3d9-77c5-432b-8487-8ca455700237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 22 20:33:41.389104 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:41.389070 2575 patch_prober.go:28] interesting pod/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 22 20:33:41.389482 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:41.389139 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" podUID="88a6e3d9-77c5-432b-8487-8ca455700237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 22 20:33:42.388444 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:42.388401 2575 patch_prober.go:28] interesting pod/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 22 20:33:42.388619 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:42.388462 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" podUID="88a6e3d9-77c5-432b-8487-8ca455700237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 22 20:33:43.392243 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:43.392211 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:43.393417 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:43.393399 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:33:49.529155 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.529121 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s"] Jul 22 20:33:49.529532 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.529393 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerName="istio-proxy" Jul 22 20:33:49.529532 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.529403 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerName="istio-proxy" Jul 22 20:33:49.529532 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.529414 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerName="istio-proxy" Jul 22 20:33:49.529532 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.529420 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerName="istio-proxy" Jul 22 20:33:49.529532 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.529495 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerName="istio-proxy" Jul 22 20:33:49.529532 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.529508 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec4fffe3-ad06-4ae1-84cf-2aa0461a37be" containerName="istio-proxy" Jul 22 20:33:49.532205 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.532180 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s" Jul 22 20:33:49.534666 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.534636 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 20:33:49.534781 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.534727 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjqt7\"" Jul 22 20:33:49.540426 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.540395 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s"] Jul 22 20:33:49.711577 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.711530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87ee689-907e-43b7-9733-5108c45c9e48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s\" (UID: \"b87ee689-907e-43b7-9733-5108c45c9e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s" Jul 22 20:33:49.813105 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.812995 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87ee689-907e-43b7-9733-5108c45c9e48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s\" (UID: \"b87ee689-907e-43b7-9733-5108c45c9e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s" Jul 22 20:33:49.813669 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.813648 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87ee689-907e-43b7-9733-5108c45c9e48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s\" (UID: \"b87ee689-907e-43b7-9733-5108c45c9e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s" Jul 22 20:33:49.843183 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.843146 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s" Jul 22 20:33:49.980852 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:49.980826 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s"] Jul 22 20:33:49.982568 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:33:49.982540 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87ee689_907e_43b7_9733_5108c45c9e48.slice/crio-b08747f4d73dadec8949320dcb364b9cc39ce84cd470d897dbc73fecb205c8b6 WatchSource:0}: Error finding container b08747f4d73dadec8949320dcb364b9cc39ce84cd470d897dbc73fecb205c8b6: Status 404 returned error can't find the container with id b08747f4d73dadec8949320dcb364b9cc39ce84cd470d897dbc73fecb205c8b6 Jul 22 20:33:50.121937 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:50.121847 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s" event={"ID":"b87ee689-907e-43b7-9733-5108c45c9e48","Type":"ContainerStarted","Data":"b08747f4d73dadec8949320dcb364b9cc39ce84cd470d897dbc73fecb205c8b6"} Jul 22 20:33:51.126864 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:51.126774 2575 generic.go:356] "Generic (PLEG): container finished" podID="b87ee689-907e-43b7-9733-5108c45c9e48" containerID="07b4f758cb0064ab792e8a2ae236fce3c4c3bb8254e35a1e3bd458dc177beb88" exitCode=0 Jul 22 20:33:51.127245 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:51.126859 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s" event={"ID":"b87ee689-907e-43b7-9733-5108c45c9e48","Type":"ContainerDied","Data":"07b4f758cb0064ab792e8a2ae236fce3c4c3bb8254e35a1e3bd458dc177beb88"} Jul 22 20:33:52.267068 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:52.267034 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s" Jul 22 20:33:52.332237 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:52.332200 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87ee689-907e-43b7-9733-5108c45c9e48-maas-api-service-ca\") pod \"b87ee689-907e-43b7-9733-5108c45c9e48\" (UID: \"b87ee689-907e-43b7-9733-5108c45c9e48\") " Jul 22 20:33:52.332562 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:52.332539 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87ee689-907e-43b7-9733-5108c45c9e48-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b87ee689-907e-43b7-9733-5108c45c9e48" (UID: "b87ee689-907e-43b7-9733-5108c45c9e48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:33:52.433283 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:52.433208 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87ee689-907e-43b7-9733-5108c45c9e48-maas-api-service-ca\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:33:53.136133 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:53.136104 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s" Jul 22 20:33:53.136133 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:53.136113 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s" event={"ID":"b87ee689-907e-43b7-9733-5108c45c9e48","Type":"ContainerDied","Data":"b08747f4d73dadec8949320dcb364b9cc39ce84cd470d897dbc73fecb205c8b6"} Jul 22 20:33:53.136372 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:53.136158 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b08747f4d73dadec8949320dcb364b9cc39ce84cd470d897dbc73fecb205c8b6" Jul 22 20:33:59.923961 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:33:59.923923 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:02.875357 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:02.875320 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:34:04.592460 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:04.592425 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s"] Jul 22 20:34:04.595221 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:04.595194 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cef58190-ffn8s"] Jul 22 20:34:04.598027 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:04.598006 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:04.733030 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:04.732998 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75"] Jul 22 20:34:04.733224 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:04.733202 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" podUID="88a6e3d9-77c5-432b-8487-8ca455700237" containerName="istio-proxy" containerID="cri-o://e175579698098c56af521fb3713426368fb994741c45005531d077295b680786" gracePeriod=30 Jul 22 20:34:04.750461 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:04.750429 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:04.986702 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:04.986656 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:05.248692 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.248600 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:05.538317 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.538243 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt"] Jul 22 20:34:05.538558 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.538543 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87ee689-907e-43b7-9733-5108c45c9e48" containerName="revoke-keys" Jul 22 20:34:05.538614 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.538560 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87ee689-907e-43b7-9733-5108c45c9e48" containerName="revoke-keys" Jul 22 20:34:05.538650 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.538624 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87ee689-907e-43b7-9733-5108c45c9e48" containerName="revoke-keys" Jul 22 20:34:05.542001 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.541965 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.544264 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.544243 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4722ebb8-openshift-default-dockercfg-kbv7r\"" Jul 22 20:34:05.553340 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.553317 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt"] Jul 22 20:34:05.635141 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.635105 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfj9b\" (UniqueName: \"kubernetes.io/projected/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-kube-api-access-bfj9b\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.635599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.635156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-token\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.635599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.635186 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-data\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.635599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.635236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-credential-socket\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.635599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.635264 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istiod-ca-cert\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.635599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.635366 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-envoy\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.635599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.635437 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-workload-certs\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.635599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.635469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-workload-socket\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.635599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.635534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-podinfo\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.736777 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.736731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-workload-certs\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.736777 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.736782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-workload-socket\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.737086 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.736832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-podinfo\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.737086 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.736855 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfj9b\" (UniqueName: \"kubernetes.io/projected/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-kube-api-access-bfj9b\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.737086 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.736880 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-token\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.737086 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.736906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-data\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.737086 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.736933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-credential-socket\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.737086 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.736961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istiod-ca-cert\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.737086 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.737028 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-envoy\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.737439 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.737226 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-workload-certs\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.737439 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.737261 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-workload-socket\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.737553 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.737479 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-data\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.737553 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.737505 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-credential-socket\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.738002 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.737955 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istiod-ca-cert\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.739279 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.739257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-envoy\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.739408 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.739391 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-podinfo\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.744561 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.744536 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfj9b\" (UniqueName: \"kubernetes.io/projected/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-kube-api-access-bfj9b\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.744667 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.744646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-token\") pod \"e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.852208 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.852124 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:05.988330 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.988259 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt"] Jul 22 20:34:05.990655 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:34:05.990627 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf5e66b_5797_4ae1_bd39_8a7a286b5727.slice/crio-4736edbc015eecf8ce7f7b2150e223f6ffed3cfda1bfd0bad31f1ee0bdbca9f5 WatchSource:0}: Error finding container 4736edbc015eecf8ce7f7b2150e223f6ffed3cfda1bfd0bad31f1ee0bdbca9f5: Status 404 returned error can't find the container with id 4736edbc015eecf8ce7f7b2150e223f6ffed3cfda1bfd0bad31f1ee0bdbca9f5 Jul 22 20:34:05.992704 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.992674 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:34:05.992766 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.992742 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:34:05.992802 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:05.992777 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:34:06.008808 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:06.008784 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b87ee689-907e-43b7-9733-5108c45c9e48" path="/var/lib/kubelet/pods/b87ee689-907e-43b7-9733-5108c45c9e48/volumes" Jul 22 20:34:06.193895 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:06.193863 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" event={"ID":"5cf5e66b-5797-4ae1-bd39-8a7a286b5727","Type":"ContainerStarted","Data":"a804a4f7061b04e39395006e5d541f71e7376901a3a771013fd21c431ba9d83a"} Jul 22 20:34:06.194149 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:06.193912 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" event={"ID":"5cf5e66b-5797-4ae1-bd39-8a7a286b5727","Type":"ContainerStarted","Data":"4736edbc015eecf8ce7f7b2150e223f6ffed3cfda1bfd0bad31f1ee0bdbca9f5"} Jul 22 20:34:06.215241 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:06.215198 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" podStartSLOduration=1.215182747 podStartE2EDuration="1.215182747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:34:06.21303593 +0000 UTC m=+1946.725915635" watchObservedRunningTime="2026-07-22 20:34:06.215182747 +0000 UTC m=+1946.728062452" Jul 22 20:34:06.853155 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:06.853121 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:07.339471 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:07.339442 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:07.390523 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:07.390489 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:07.857602 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:07.857568 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:08.201848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:08.201823 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:08.202686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:08.202662 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:09.289465 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:09.289428 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:09.390259 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:09.390225 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:09.981381 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:09.981353 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:34:10.076848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.076747 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a6e3d9-77c5-432b-8487-8ca455700237-istio-podinfo\") pod \"88a6e3d9-77c5-432b-8487-8ca455700237\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " Jul 22 20:34:10.076848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.076804 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-istio-envoy\") pod \"88a6e3d9-77c5-432b-8487-8ca455700237\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " Jul 22 20:34:10.076848 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.076827 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7xqr\" (UniqueName: \"kubernetes.io/projected/88a6e3d9-77c5-432b-8487-8ca455700237-kube-api-access-t7xqr\") pod \"88a6e3d9-77c5-432b-8487-8ca455700237\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " Jul 22 20:34:10.077160 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.076859 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-workload-certs\") pod \"88a6e3d9-77c5-432b-8487-8ca455700237\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " Jul 22 20:34:10.077160 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.076881 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a6e3d9-77c5-432b-8487-8ca455700237-istiod-ca-cert\") pod \"88a6e3d9-77c5-432b-8487-8ca455700237\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " Jul 22 20:34:10.077160 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.077007 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a6e3d9-77c5-432b-8487-8ca455700237-istio-token\") pod \"88a6e3d9-77c5-432b-8487-8ca455700237\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " Jul 22 20:34:10.077160 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.077068 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-workload-socket\") pod \"88a6e3d9-77c5-432b-8487-8ca455700237\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " Jul 22 20:34:10.077160 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.077139 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-credential-socket\") pod \"88a6e3d9-77c5-432b-8487-8ca455700237\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " Jul 22 20:34:10.077398 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.077169 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-istio-data\") pod \"88a6e3d9-77c5-432b-8487-8ca455700237\" (UID: \"88a6e3d9-77c5-432b-8487-8ca455700237\") " Jul 22 20:34:10.077398 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.077234 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88a6e3d9-77c5-432b-8487-8ca455700237" (UID: "88a6e3d9-77c5-432b-8487-8ca455700237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:34:10.077398 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.077336 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a6e3d9-77c5-432b-8487-8ca455700237-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88a6e3d9-77c5-432b-8487-8ca455700237" (UID: "88a6e3d9-77c5-432b-8487-8ca455700237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:34:10.077599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.077578 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-workload-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:10.077725 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.077605 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a6e3d9-77c5-432b-8487-8ca455700237-istiod-ca-cert\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:10.077725 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.077586 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88a6e3d9-77c5-432b-8487-8ca455700237" (UID: "88a6e3d9-77c5-432b-8487-8ca455700237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:34:10.077829 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.077747 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88a6e3d9-77c5-432b-8487-8ca455700237" (UID: "88a6e3d9-77c5-432b-8487-8ca455700237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:34:10.078258 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.078230 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-istio-data" (OuterVolumeSpecName: "istio-data") pod "88a6e3d9-77c5-432b-8487-8ca455700237" (UID: "88a6e3d9-77c5-432b-8487-8ca455700237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:34:10.079591 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.079573 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88a6e3d9-77c5-432b-8487-8ca455700237-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88a6e3d9-77c5-432b-8487-8ca455700237" (UID: "88a6e3d9-77c5-432b-8487-8ca455700237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 20:34:10.079686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.079610 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a6e3d9-77c5-432b-8487-8ca455700237-kube-api-access-t7xqr" (OuterVolumeSpecName: "kube-api-access-t7xqr") pod "88a6e3d9-77c5-432b-8487-8ca455700237" (UID: "88a6e3d9-77c5-432b-8487-8ca455700237"). InnerVolumeSpecName "kube-api-access-t7xqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:34:10.079686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.079610 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a6e3d9-77c5-432b-8487-8ca455700237-istio-token" (OuterVolumeSpecName: "istio-token") pod "88a6e3d9-77c5-432b-8487-8ca455700237" (UID: "88a6e3d9-77c5-432b-8487-8ca455700237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:34:10.079785 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.079714 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88a6e3d9-77c5-432b-8487-8ca455700237" (UID: "88a6e3d9-77c5-432b-8487-8ca455700237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:34:10.178171 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.178137 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a6e3d9-77c5-432b-8487-8ca455700237-istio-podinfo\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:10.178171 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.178165 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-istio-envoy\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:10.178171 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.178175 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7xqr\" (UniqueName: \"kubernetes.io/projected/88a6e3d9-77c5-432b-8487-8ca455700237-kube-api-access-t7xqr\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:10.178382 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.178185 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a6e3d9-77c5-432b-8487-8ca455700237-istio-token\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:10.178382 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.178193 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-workload-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:10.178382 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.178201 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-credential-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:10.178382 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.178209 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a6e3d9-77c5-432b-8487-8ca455700237-istio-data\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:10.211993 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.211940 2575 generic.go:356] "Generic (PLEG): container finished" podID="88a6e3d9-77c5-432b-8487-8ca455700237" containerID="e175579698098c56af521fb3713426368fb994741c45005531d077295b680786" exitCode=0 Jul 22 20:34:10.212124 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.212011 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" Jul 22 20:34:10.212124 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.212010 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" event={"ID":"88a6e3d9-77c5-432b-8487-8ca455700237","Type":"ContainerDied","Data":"e175579698098c56af521fb3713426368fb994741c45005531d077295b680786"} Jul 22 20:34:10.212124 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.212069 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75" event={"ID":"88a6e3d9-77c5-432b-8487-8ca455700237","Type":"ContainerDied","Data":"ad5359ac111c96a8e57034b7fe7df50de31d64e04f9add18bf0b3e54d88ecb76"} Jul 22 20:34:10.212124 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.212085 2575 scope.go:117] "RemoveContainer" containerID="e175579698098c56af521fb3713426368fb994741c45005531d077295b680786" Jul 22 20:34:10.220379 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.220363 2575 scope.go:117] "RemoveContainer" containerID="5d879f9c94f124b302057e56515313c73dc5956502ce96fc3227443ddfe7d494" Jul 22 20:34:10.228304 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.228283 2575 scope.go:117] "RemoveContainer" containerID="e175579698098c56af521fb3713426368fb994741c45005531d077295b680786" Jul 22 20:34:10.228558 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:34:10.228541 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e175579698098c56af521fb3713426368fb994741c45005531d077295b680786\": container with ID starting with e175579698098c56af521fb3713426368fb994741c45005531d077295b680786 not found: ID does not exist" containerID="e175579698098c56af521fb3713426368fb994741c45005531d077295b680786" Jul 22 20:34:10.228644 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.228566 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e175579698098c56af521fb3713426368fb994741c45005531d077295b680786"} err="failed to get container status \"e175579698098c56af521fb3713426368fb994741c45005531d077295b680786\": rpc error: code = NotFound desc = could not find container \"e175579698098c56af521fb3713426368fb994741c45005531d077295b680786\": container with ID starting with e175579698098c56af521fb3713426368fb994741c45005531d077295b680786 not found: ID does not exist" Jul 22 20:34:10.228644 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.228584 2575 scope.go:117] "RemoveContainer" containerID="5d879f9c94f124b302057e56515313c73dc5956502ce96fc3227443ddfe7d494" Jul 22 20:34:10.228808 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:34:10.228791 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d879f9c94f124b302057e56515313c73dc5956502ce96fc3227443ddfe7d494\": container with ID starting with 5d879f9c94f124b302057e56515313c73dc5956502ce96fc3227443ddfe7d494 not found: ID does not exist" containerID="5d879f9c94f124b302057e56515313c73dc5956502ce96fc3227443ddfe7d494" Jul 22 20:34:10.228845 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.228814 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d879f9c94f124b302057e56515313c73dc5956502ce96fc3227443ddfe7d494"} err="failed to get container status \"5d879f9c94f124b302057e56515313c73dc5956502ce96fc3227443ddfe7d494\": rpc error: code = NotFound desc = could not find container \"5d879f9c94f124b302057e56515313c73dc5956502ce96fc3227443ddfe7d494\": container with ID starting with 5d879f9c94f124b302057e56515313c73dc5956502ce96fc3227443ddfe7d494 not found: ID does not exist" Jul 22 20:34:10.233925 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.233895 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75"] Jul 22 20:34:10.240008 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:10.239966 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cef58190-gw-openshift-default-57cb67cb8c-prq75"] Jul 22 20:34:12.006266 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:12.006239 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88a6e3d9-77c5-432b-8487-8ca455700237" path="/var/lib/kubelet/pods/88a6e3d9-77c5-432b-8487-8ca455700237/volumes" Jul 22 20:34:13.964275 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:13.964243 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:34:22.266182 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:22.266144 2575 generic.go:356] "Generic (PLEG): container finished" podID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" containerID="a804a4f7061b04e39395006e5d541f71e7376901a3a771013fd21c431ba9d83a" exitCode=0 Jul 22 20:34:22.266604 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:22.266223 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" event={"ID":"5cf5e66b-5797-4ae1-bd39-8a7a286b5727","Type":"ContainerDied","Data":"a804a4f7061b04e39395006e5d541f71e7376901a3a771013fd21c431ba9d83a"} Jul 22 20:34:22.266647 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:22.266615 2575 scope.go:117] "RemoveContainer" containerID="a804a4f7061b04e39395006e5d541f71e7376901a3a771013fd21c431ba9d83a" Jul 22 20:34:22.267638 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:22.267610 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:34:22.267711 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:22.267694 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:34:22.267761 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:22.267744 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:34:23.202675 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:23.202616 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:23.272558 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:23.272526 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" event={"ID":"5cf5e66b-5797-4ae1-bd39-8a7a286b5727","Type":"ContainerStarted","Data":"3124303e967e955e69b172a4e1e3101bdc149a2081ee1cbdb5020c976acea303"} Jul 22 20:34:23.852998 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:23.852947 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:23.853916 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:23.853890 2575 patch_prober.go:28] interesting pod/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 22 20:34:23.854040 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:23.853939 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" podUID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 22 20:34:24.852498 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:24.852458 2575 patch_prober.go:28] interesting pod/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 22 20:34:24.852875 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:24.852529 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" podUID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 22 20:34:25.852824 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:25.852785 2575 patch_prober.go:28] interesting pod/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 22 20:34:25.853222 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:25.852842 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" podUID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 22 20:34:26.852958 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:26.852922 2575 patch_prober.go:28] interesting pod/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 22 20:34:26.853363 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:26.853033 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" podUID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 22 20:34:27.856475 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:27.856444 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:27.856887 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:27.856806 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:27.857599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:27.857562 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:33.396603 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.396523 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n"] Jul 22 20:34:33.396948 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.396936 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a6e3d9-77c5-432b-8487-8ca455700237" containerName="istio-proxy" Jul 22 20:34:33.397008 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.396954 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a6e3d9-77c5-432b-8487-8ca455700237" containerName="istio-proxy" Jul 22 20:34:33.397008 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.396992 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a6e3d9-77c5-432b-8487-8ca455700237" containerName="istio-proxy" Jul 22 20:34:33.397008 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.397001 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a6e3d9-77c5-432b-8487-8ca455700237" containerName="istio-proxy" Jul 22 20:34:33.397102 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.397074 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a6e3d9-77c5-432b-8487-8ca455700237" containerName="istio-proxy" Jul 22 20:34:33.399844 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.399826 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n" Jul 22 20:34:33.402342 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.402324 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjqt7\"" Jul 22 20:34:33.402452 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.402401 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 20:34:33.406181 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.406158 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n"] Jul 22 20:34:33.590130 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.590078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbd4b47-77ad-46ae-a74e-8a365ffb8065-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n\" (UID: \"0dbd4b47-77ad-46ae-a74e-8a365ffb8065\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n" Jul 22 20:34:33.691205 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.691179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbd4b47-77ad-46ae-a74e-8a365ffb8065-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n\" (UID: \"0dbd4b47-77ad-46ae-a74e-8a365ffb8065\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n" Jul 22 20:34:33.691777 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.691756 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbd4b47-77ad-46ae-a74e-8a365ffb8065-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n\" (UID: \"0dbd4b47-77ad-46ae-a74e-8a365ffb8065\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n" Jul 22 20:34:33.713225 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:33.713200 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n" Jul 22 20:34:34.043121 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:34.043095 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n"] Jul 22 20:34:34.045355 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:34:34.045331 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbd4b47_77ad_46ae_a74e_8a365ffb8065.slice/crio-0f5db2492badb339d593065adb3e7e4aabc32c4c1e98f7d3d772625c350d4b86 WatchSource:0}: Error finding container 0f5db2492badb339d593065adb3e7e4aabc32c4c1e98f7d3d772625c350d4b86: Status 404 returned error can't find the container with id 0f5db2492badb339d593065adb3e7e4aabc32c4c1e98f7d3d772625c350d4b86 Jul 22 20:34:34.319245 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:34.319164 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n" event={"ID":"0dbd4b47-77ad-46ae-a74e-8a365ffb8065","Type":"ContainerStarted","Data":"0bd60622d9f2df2aece829b9d4837dc37bfdbce94f582d8fb179c72ca509c067"} Jul 22 20:34:34.319245 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:34.319214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n" event={"ID":"0dbd4b47-77ad-46ae-a74e-8a365ffb8065","Type":"ContainerStarted","Data":"0f5db2492badb339d593065adb3e7e4aabc32c4c1e98f7d3d772625c350d4b86"} Jul 22 20:34:34.334462 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:34.334401 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n" podStartSLOduration=1.334386832 podStartE2EDuration="1.334386832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:34:34.333313046 +0000 UTC m=+1974.846192751" watchObservedRunningTime="2026-07-22 20:34:34.334386832 +0000 UTC m=+1974.847266570" Jul 22 20:34:35.324793 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:35.324764 2575 generic.go:356] "Generic (PLEG): container finished" podID="0dbd4b47-77ad-46ae-a74e-8a365ffb8065" containerID="0bd60622d9f2df2aece829b9d4837dc37bfdbce94f582d8fb179c72ca509c067" exitCode=0 Jul 22 20:34:35.325173 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:35.324846 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n" event={"ID":"0dbd4b47-77ad-46ae-a74e-8a365ffb8065","Type":"ContainerDied","Data":"0bd60622d9f2df2aece829b9d4837dc37bfdbce94f582d8fb179c72ca509c067"} Jul 22 20:34:36.456126 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:36.456103 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n" Jul 22 20:34:36.513387 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:36.513361 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbd4b47-77ad-46ae-a74e-8a365ffb8065-maas-api-service-ca\") pod \"0dbd4b47-77ad-46ae-a74e-8a365ffb8065\" (UID: \"0dbd4b47-77ad-46ae-a74e-8a365ffb8065\") " Jul 22 20:34:36.513704 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:36.513681 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbd4b47-77ad-46ae-a74e-8a365ffb8065-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dbd4b47-77ad-46ae-a74e-8a365ffb8065" (UID: "0dbd4b47-77ad-46ae-a74e-8a365ffb8065"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:34:36.613906 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:36.613836 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbd4b47-77ad-46ae-a74e-8a365ffb8065-maas-api-service-ca\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:37.334696 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:37.334662 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n" Jul 22 20:34:37.334696 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:37.334667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n" event={"ID":"0dbd4b47-77ad-46ae-a74e-8a365ffb8065","Type":"ContainerDied","Data":"0f5db2492badb339d593065adb3e7e4aabc32c4c1e98f7d3d772625c350d4b86"} Jul 22 20:34:37.335023 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:37.334720 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f5db2492badb339d593065adb3e7e4aabc32c4c1e98f7d3d772625c350d4b86" Jul 22 20:34:43.826444 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:43.826404 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:46.773686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:46.773649 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:34:48.477210 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:48.477180 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n"] Jul 22 20:34:48.482527 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:48.482491 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4722ebb8-gw96n"] Jul 22 20:34:48.483256 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:48.483233 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:49.687614 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:49.687575 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt"] Jul 22 20:34:49.688045 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:49.687818 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" podUID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" containerName="istio-proxy" containerID="cri-o://3124303e967e955e69b172a4e1e3101bdc149a2081ee1cbdb5020c976acea303" gracePeriod=30 Jul 22 20:34:49.733686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:49.733649 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:49.926092 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:49.926060 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:50.006763 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.006689 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dbd4b47-77ad-46ae-a74e-8a365ffb8065" path="/var/lib/kubelet/pods/0dbd4b47-77ad-46ae-a74e-8a365ffb8065/volumes" Jul 22 20:34:50.151635 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.151599 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:50.454312 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.454282 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh"] Jul 22 20:34:50.454571 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.454561 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dbd4b47-77ad-46ae-a74e-8a365ffb8065" containerName="revoke-keys" Jul 22 20:34:50.454617 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.454573 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dbd4b47-77ad-46ae-a74e-8a365ffb8065" containerName="revoke-keys" Jul 22 20:34:50.454651 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.454644 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dbd4b47-77ad-46ae-a74e-8a365ffb8065" containerName="revoke-keys" Jul 22 20:34:50.454684 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.454652 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a6e3d9-77c5-432b-8487-8ca455700237" containerName="istio-proxy" Jul 22 20:34:50.457482 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.457467 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.460549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.460529 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6627cefd-openshift-default-dockercfg-cjq2s\"" Jul 22 20:34:50.469250 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.469225 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh"] Jul 22 20:34:50.481292 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.481265 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:50.521516 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.521487 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51091562-0995-47b8-b010-83df9483eb9e-istio-podinfo\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.521715 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.521523 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxts\" (UniqueName: \"kubernetes.io/projected/51091562-0995-47b8-b010-83df9483eb9e-kube-api-access-fxxts\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.521715 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.521571 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-istio-data\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.521715 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.521646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-workload-certs\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.521715 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.521670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51091562-0995-47b8-b010-83df9483eb9e-istiod-ca-cert\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.521894 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.521736 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51091562-0995-47b8-b010-83df9483eb9e-istio-token\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.521894 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.521787 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-istio-envoy\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.521894 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.521839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-workload-socket\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.521894 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.521868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-credential-socket\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.622931 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.622895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-istio-data\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.623149 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.622951 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-workload-certs\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.623149 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.623035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51091562-0995-47b8-b010-83df9483eb9e-istiod-ca-cert\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.623149 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.623113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51091562-0995-47b8-b010-83df9483eb9e-istio-token\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.623315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.623172 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-istio-envoy\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.623315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.623202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-workload-socket\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.623315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.623225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-credential-socket\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.623315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.623260 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51091562-0995-47b8-b010-83df9483eb9e-istio-podinfo\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.623315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.623284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxts\" (UniqueName: \"kubernetes.io/projected/51091562-0995-47b8-b010-83df9483eb9e-kube-api-access-fxxts\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.623599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.623328 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-workload-certs\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.623599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.623551 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-credential-socket\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.623599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.623573 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-workload-socket\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.623739 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.623659 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-istio-data\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.623739 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.623712 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51091562-0995-47b8-b010-83df9483eb9e-istiod-ca-cert\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.625380 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.625352 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-istio-envoy\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.625580 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.625564 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51091562-0995-47b8-b010-83df9483eb9e-istio-podinfo\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.630921 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.630897 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51091562-0995-47b8-b010-83df9483eb9e-istio-token\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.631222 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.631196 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxts\" (UniqueName: \"kubernetes.io/projected/51091562-0995-47b8-b010-83df9483eb9e-kube-api-access-fxxts\") pod \"e2e-derive-6627cefd-openshift-default-8565f58855-8gldh\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.758922 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.758834 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:50.767086 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.767067 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:50.904964 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.904930 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh"] Jul 22 20:34:50.912239 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:34:50.912213 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51091562_0995_47b8_b010_83df9483eb9e.slice/crio-3e87925679d3d75b19ac81b4c51043228110f2ab9c184c1de6f82c00e3e50e61 WatchSource:0}: Error finding container 3e87925679d3d75b19ac81b4c51043228110f2ab9c184c1de6f82c00e3e50e61: Status 404 returned error can't find the container with id 3e87925679d3d75b19ac81b4c51043228110f2ab9c184c1de6f82c00e3e50e61 Jul 22 20:34:50.914226 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.914193 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:34:50.914306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.914256 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:34:50.914306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:50.914283 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:34:51.395382 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:51.395348 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" event={"ID":"51091562-0995-47b8-b010-83df9483eb9e","Type":"ContainerStarted","Data":"f6d23bbb8473dd09821dbb8f94dc108f7481be3f0f93c2fa218221d615b3de70"} Jul 22 20:34:51.395550 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:51.395403 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" event={"ID":"51091562-0995-47b8-b010-83df9483eb9e","Type":"ContainerStarted","Data":"3e87925679d3d75b19ac81b4c51043228110f2ab9c184c1de6f82c00e3e50e61"} Jul 22 20:34:51.415075 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:51.415003 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" podStartSLOduration=1.414964429 podStartE2EDuration="1.414964429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:34:51.414160339 +0000 UTC m=+1991.927040085" watchObservedRunningTime="2026-07-22 20:34:51.414964429 +0000 UTC m=+1991.927844132" Jul 22 20:34:51.768313 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:51.768270 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:51.773107 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:51.773081 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:52.399715 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:52.399684 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:52.400537 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:52.400518 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:34:52.753211 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:52.753171 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:52.845806 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:52.845772 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:52.915984 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:52.915931 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:53.270402 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:53.270372 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:53.494818 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:53.494772 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:53.597621 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:53.597483 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:34:54.928505 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.928481 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:54.963158 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.963082 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-workload-socket\") pod \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " Jul 22 20:34:54.963158 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.963141 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-envoy\") pod \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " Jul 22 20:34:54.963360 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.963188 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfj9b\" (UniqueName: \"kubernetes.io/projected/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-kube-api-access-bfj9b\") pod \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " Jul 22 20:34:54.963360 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.963219 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-workload-certs\") pod \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " Jul 22 20:34:54.963360 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.963255 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-token\") pod \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " Jul 22 20:34:54.963360 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.963281 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-podinfo\") pod \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " Jul 22 20:34:54.963360 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.963306 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-credential-socket\") pod \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " Jul 22 20:34:54.963360 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.963342 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istiod-ca-cert\") pod \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " Jul 22 20:34:54.963644 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.963371 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5cf5e66b-5797-4ae1-bd39-8a7a286b5727" (UID: "5cf5e66b-5797-4ae1-bd39-8a7a286b5727"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:34:54.963644 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.963418 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-data\") pod \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\" (UID: \"5cf5e66b-5797-4ae1-bd39-8a7a286b5727\") " Jul 22 20:34:54.963748 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.963674 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-workload-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:54.963957 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.963802 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5cf5e66b-5797-4ae1-bd39-8a7a286b5727" (UID: "5cf5e66b-5797-4ae1-bd39-8a7a286b5727"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:34:54.963957 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.963814 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5cf5e66b-5797-4ae1-bd39-8a7a286b5727" (UID: "5cf5e66b-5797-4ae1-bd39-8a7a286b5727"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:34:54.964120 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.964041 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5cf5e66b-5797-4ae1-bd39-8a7a286b5727" (UID: "5cf5e66b-5797-4ae1-bd39-8a7a286b5727"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:34:54.964506 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.964476 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-data" (OuterVolumeSpecName: "istio-data") pod "5cf5e66b-5797-4ae1-bd39-8a7a286b5727" (UID: "5cf5e66b-5797-4ae1-bd39-8a7a286b5727"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:34:54.965803 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.965755 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5cf5e66b-5797-4ae1-bd39-8a7a286b5727" (UID: "5cf5e66b-5797-4ae1-bd39-8a7a286b5727"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:34:54.965919 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.965830 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-kube-api-access-bfj9b" (OuterVolumeSpecName: "kube-api-access-bfj9b") pod "5cf5e66b-5797-4ae1-bd39-8a7a286b5727" (UID: "5cf5e66b-5797-4ae1-bd39-8a7a286b5727"). InnerVolumeSpecName "kube-api-access-bfj9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:34:54.965919 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.965831 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-token" (OuterVolumeSpecName: "istio-token") pod "5cf5e66b-5797-4ae1-bd39-8a7a286b5727" (UID: "5cf5e66b-5797-4ae1-bd39-8a7a286b5727"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:34:54.965919 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:54.965901 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5cf5e66b-5797-4ae1-bd39-8a7a286b5727" (UID: "5cf5e66b-5797-4ae1-bd39-8a7a286b5727"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 20:34:55.064954 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.064917 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istiod-ca-cert\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:55.064954 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.064948 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-data\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:55.064954 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.064956 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-envoy\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:55.064954 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.064964 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfj9b\" (UniqueName: \"kubernetes.io/projected/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-kube-api-access-bfj9b\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:55.065231 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.064995 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-workload-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:55.065231 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.065004 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-token\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:55.065231 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.065012 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-istio-podinfo\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:55.065231 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.065020 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5e66b-5797-4ae1-bd39-8a7a286b5727-credential-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:34:55.413487 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.413396 2575 generic.go:356] "Generic (PLEG): container finished" podID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" containerID="3124303e967e955e69b172a4e1e3101bdc149a2081ee1cbdb5020c976acea303" exitCode=0 Jul 22 20:34:55.413487 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.413447 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" Jul 22 20:34:55.413698 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.413482 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" event={"ID":"5cf5e66b-5797-4ae1-bd39-8a7a286b5727","Type":"ContainerDied","Data":"3124303e967e955e69b172a4e1e3101bdc149a2081ee1cbdb5020c976acea303"} Jul 22 20:34:55.413698 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.413558 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt" event={"ID":"5cf5e66b-5797-4ae1-bd39-8a7a286b5727","Type":"ContainerDied","Data":"4736edbc015eecf8ce7f7b2150e223f6ffed3cfda1bfd0bad31f1ee0bdbca9f5"} Jul 22 20:34:55.413698 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.413606 2575 scope.go:117] "RemoveContainer" containerID="3124303e967e955e69b172a4e1e3101bdc149a2081ee1cbdb5020c976acea303" Jul 22 20:34:55.422259 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.422243 2575 scope.go:117] "RemoveContainer" containerID="a804a4f7061b04e39395006e5d541f71e7376901a3a771013fd21c431ba9d83a" Jul 22 20:34:55.429565 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.429549 2575 scope.go:117] "RemoveContainer" containerID="3124303e967e955e69b172a4e1e3101bdc149a2081ee1cbdb5020c976acea303" Jul 22 20:34:55.429816 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:34:55.429796 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3124303e967e955e69b172a4e1e3101bdc149a2081ee1cbdb5020c976acea303\": container with ID starting with 3124303e967e955e69b172a4e1e3101bdc149a2081ee1cbdb5020c976acea303 not found: ID does not exist" containerID="3124303e967e955e69b172a4e1e3101bdc149a2081ee1cbdb5020c976acea303" Jul 22 20:34:55.429880 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.429825 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3124303e967e955e69b172a4e1e3101bdc149a2081ee1cbdb5020c976acea303"} err="failed to get container status \"3124303e967e955e69b172a4e1e3101bdc149a2081ee1cbdb5020c976acea303\": rpc error: code = NotFound desc = could not find container \"3124303e967e955e69b172a4e1e3101bdc149a2081ee1cbdb5020c976acea303\": container with ID starting with 3124303e967e955e69b172a4e1e3101bdc149a2081ee1cbdb5020c976acea303 not found: ID does not exist" Jul 22 20:34:55.429880 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.429843 2575 scope.go:117] "RemoveContainer" containerID="a804a4f7061b04e39395006e5d541f71e7376901a3a771013fd21c431ba9d83a" Jul 22 20:34:55.430141 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:34:55.430120 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a804a4f7061b04e39395006e5d541f71e7376901a3a771013fd21c431ba9d83a\": container with ID starting with a804a4f7061b04e39395006e5d541f71e7376901a3a771013fd21c431ba9d83a not found: ID does not exist" containerID="a804a4f7061b04e39395006e5d541f71e7376901a3a771013fd21c431ba9d83a" Jul 22 20:34:55.430187 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.430147 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a804a4f7061b04e39395006e5d541f71e7376901a3a771013fd21c431ba9d83a"} err="failed to get container status \"a804a4f7061b04e39395006e5d541f71e7376901a3a771013fd21c431ba9d83a\": rpc error: code = NotFound desc = could not find container \"a804a4f7061b04e39395006e5d541f71e7376901a3a771013fd21c431ba9d83a\": container with ID starting with a804a4f7061b04e39395006e5d541f71e7376901a3a771013fd21c431ba9d83a not found: ID does not exist" Jul 22 20:34:55.434458 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.434436 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt"] Jul 22 20:34:55.438351 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:55.438329 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4722ebb8-openshift-default-58d796b87f-bgzjt"] Jul 22 20:34:56.007157 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:56.007127 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" path="/var/lib/kubelet/pods/5cf5e66b-5797-4ae1-bd39-8a7a286b5727/volumes" Jul 22 20:34:57.986878 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:34:57.986848 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:35:05.595775 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:05.595730 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:05.693173 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:05.693135 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:06.461850 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:06.461818 2575 generic.go:356] "Generic (PLEG): container finished" podID="51091562-0995-47b8-b010-83df9483eb9e" containerID="f6d23bbb8473dd09821dbb8f94dc108f7481be3f0f93c2fa218221d615b3de70" exitCode=0 Jul 22 20:35:06.462038 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:06.461893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" event={"ID":"51091562-0995-47b8-b010-83df9483eb9e","Type":"ContainerDied","Data":"f6d23bbb8473dd09821dbb8f94dc108f7481be3f0f93c2fa218221d615b3de70"} Jul 22 20:35:06.462302 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:06.462288 2575 scope.go:117] "RemoveContainer" containerID="f6d23bbb8473dd09821dbb8f94dc108f7481be3f0f93c2fa218221d615b3de70" Jul 22 20:35:06.463298 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:06.463269 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:35:06.463369 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:06.463353 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:35:06.463434 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:06.463421 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:35:07.400636 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:07.400524 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:35:07.467387 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:07.467353 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" event={"ID":"51091562-0995-47b8-b010-83df9483eb9e","Type":"ContainerStarted","Data":"b9493f7b4c1fd4a48b949225a1ad0e7431eaf15341a76a19cad635e7bac3d48a"} Jul 22 20:35:07.767746 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:07.767705 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:35:07.768770 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:07.768743 2575 patch_prober.go:28] interesting pod/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 22 20:35:07.768869 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:07.768793 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" podUID="51091562-0995-47b8-b010-83df9483eb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 22 20:35:08.768362 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:08.768328 2575 patch_prober.go:28] interesting pod/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 22 20:35:08.768762 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:08.768385 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" podUID="51091562-0995-47b8-b010-83df9483eb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 22 20:35:09.767790 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:09.767757 2575 patch_prober.go:28] interesting pod/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 22 20:35:09.768010 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:09.767813 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" podUID="51091562-0995-47b8-b010-83df9483eb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 22 20:35:10.767465 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:10.767429 2575 patch_prober.go:28] interesting pod/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 22 20:35:10.767854 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:10.767483 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" podUID="51091562-0995-47b8-b010-83df9483eb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 22 20:35:11.771931 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:11.771899 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:35:11.772292 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:11.772262 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:35:11.772924 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:11.772903 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:35:13.715173 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:13.715134 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:14.293065 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:14.293029 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:14.396097 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:14.396058 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:17.681753 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:17.681717 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk"] Jul 22 20:35:17.682142 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:17.681993 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" containerName="istio-proxy" Jul 22 20:35:17.682142 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:17.682004 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" containerName="istio-proxy" Jul 22 20:35:17.682142 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:17.682068 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" containerName="istio-proxy" Jul 22 20:35:17.683737 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:17.683721 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk" Jul 22 20:35:17.685885 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:17.685859 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 20:35:17.686154 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:17.686140 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjqt7\"" Jul 22 20:35:17.691736 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:17.691714 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk"] Jul 22 20:35:17.768097 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:17.768065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c740631-a06b-472c-bc75-2e9920a71125-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk\" (UID: \"0c740631-a06b-472c-bc75-2e9920a71125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk" Jul 22 20:35:17.869200 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:17.869167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c740631-a06b-472c-bc75-2e9920a71125-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk\" (UID: \"0c740631-a06b-472c-bc75-2e9920a71125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk" Jul 22 20:35:17.869885 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:17.869861 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c740631-a06b-472c-bc75-2e9920a71125-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk\" (UID: \"0c740631-a06b-472c-bc75-2e9920a71125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk" Jul 22 20:35:17.994583 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:17.994551 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk" Jul 22 20:35:18.120743 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:18.120718 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk"] Jul 22 20:35:18.122909 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:35:18.122878 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c740631_a06b_472c_bc75_2e9920a71125.slice/crio-c3f6af99e7f70cbe3dd12a85d740e95ccf11047328be5d2107a8a85789354d5a WatchSource:0}: Error finding container c3f6af99e7f70cbe3dd12a85d740e95ccf11047328be5d2107a8a85789354d5a: Status 404 returned error can't find the container with id c3f6af99e7f70cbe3dd12a85d740e95ccf11047328be5d2107a8a85789354d5a Jul 22 20:35:18.514511 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:18.514477 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk" event={"ID":"0c740631-a06b-472c-bc75-2e9920a71125","Type":"ContainerStarted","Data":"366930324caed8965f54d1182ba27e62d6bdf1ed0900901ab7fc1680e91ea1db"} Jul 22 20:35:18.514699 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:18.514533 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk" event={"ID":"0c740631-a06b-472c-bc75-2e9920a71125","Type":"ContainerStarted","Data":"c3f6af99e7f70cbe3dd12a85d740e95ccf11047328be5d2107a8a85789354d5a"} Jul 22 20:35:18.529355 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:18.529311 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk" podStartSLOduration=1.529296242 podStartE2EDuration="1.529296242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:35:18.527391681 +0000 UTC m=+2019.040271386" watchObservedRunningTime="2026-07-22 20:35:18.529296242 +0000 UTC m=+2019.042175980" Jul 22 20:35:19.520202 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:19.520168 2575 generic.go:356] "Generic (PLEG): container finished" podID="0c740631-a06b-472c-bc75-2e9920a71125" containerID="366930324caed8965f54d1182ba27e62d6bdf1ed0900901ab7fc1680e91ea1db" exitCode=0 Jul 22 20:35:19.520637 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:19.520233 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk" event={"ID":"0c740631-a06b-472c-bc75-2e9920a71125","Type":"ContainerDied","Data":"366930324caed8965f54d1182ba27e62d6bdf1ed0900901ab7fc1680e91ea1db"} Jul 22 20:35:20.648043 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:20.648018 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk" Jul 22 20:35:20.793557 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:20.793474 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c740631-a06b-472c-bc75-2e9920a71125-maas-api-service-ca\") pod \"0c740631-a06b-472c-bc75-2e9920a71125\" (UID: \"0c740631-a06b-472c-bc75-2e9920a71125\") " Jul 22 20:35:20.793830 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:20.793809 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c740631-a06b-472c-bc75-2e9920a71125-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0c740631-a06b-472c-bc75-2e9920a71125" (UID: "0c740631-a06b-472c-bc75-2e9920a71125"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:35:20.894027 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:20.893997 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c740631-a06b-472c-bc75-2e9920a71125-maas-api-service-ca\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:35:21.530631 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:21.530598 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk" event={"ID":"0c740631-a06b-472c-bc75-2e9920a71125","Type":"ContainerDied","Data":"c3f6af99e7f70cbe3dd12a85d740e95ccf11047328be5d2107a8a85789354d5a"} Jul 22 20:35:21.530807 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:21.530642 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3f6af99e7f70cbe3dd12a85d740e95ccf11047328be5d2107a8a85789354d5a" Jul 22 20:35:21.530807 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:21.530606 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk" Jul 22 20:35:28.150867 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:28.150828 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:31.279041 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:31.279004 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:35:32.760991 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:32.760952 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk"] Jul 22 20:35:32.763157 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:32.763130 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6627cefd-kjhtk"] Jul 22 20:35:32.764159 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:32.764129 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:32.894427 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:32.894397 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh"] Jul 22 20:35:32.894624 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:32.894601 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" podUID="51091562-0995-47b8-b010-83df9483eb9e" containerName="istio-proxy" containerID="cri-o://b9493f7b4c1fd4a48b949225a1ad0e7431eaf15341a76a19cad635e7bac3d48a" gracePeriod=30 Jul 22 20:35:32.918123 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:32.918095 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:33.144938 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:33.144851 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:33.657510 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:33.657473 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:33.740922 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:33.740891 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:33.840934 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:33.840898 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:34.006048 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:34.006020 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c740631-a06b-472c-bc75-2e9920a71125" path="/var/lib/kubelet/pods/0c740631-a06b-472c-bc75-2e9920a71125/volumes" Jul 22 20:35:35.238088 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:35.238059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:38.152314 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.152292 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:35:38.234214 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234179 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-workload-certs\") pod \"51091562-0995-47b8-b010-83df9483eb9e\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " Jul 22 20:35:38.234396 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234229 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-istio-envoy\") pod \"51091562-0995-47b8-b010-83df9483eb9e\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " Jul 22 20:35:38.234396 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234251 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-credential-socket\") pod \"51091562-0995-47b8-b010-83df9483eb9e\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " Jul 22 20:35:38.234396 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234269 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-istio-data\") pod \"51091562-0995-47b8-b010-83df9483eb9e\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " Jul 22 20:35:38.234396 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234328 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxxts\" (UniqueName: \"kubernetes.io/projected/51091562-0995-47b8-b010-83df9483eb9e-kube-api-access-fxxts\") pod \"51091562-0995-47b8-b010-83df9483eb9e\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " Jul 22 20:35:38.234396 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234343 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-workload-socket\") pod \"51091562-0995-47b8-b010-83df9483eb9e\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " Jul 22 20:35:38.234396 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234369 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51091562-0995-47b8-b010-83df9483eb9e-istiod-ca-cert\") pod \"51091562-0995-47b8-b010-83df9483eb9e\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " Jul 22 20:35:38.234709 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234404 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51091562-0995-47b8-b010-83df9483eb9e-istio-token\") pod \"51091562-0995-47b8-b010-83df9483eb9e\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " Jul 22 20:35:38.234709 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234432 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51091562-0995-47b8-b010-83df9483eb9e-istio-podinfo\") pod \"51091562-0995-47b8-b010-83df9483eb9e\" (UID: \"51091562-0995-47b8-b010-83df9483eb9e\") " Jul 22 20:35:38.234709 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234460 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51091562-0995-47b8-b010-83df9483eb9e" (UID: "51091562-0995-47b8-b010-83df9483eb9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:35:38.234709 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234529 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51091562-0995-47b8-b010-83df9483eb9e" (UID: "51091562-0995-47b8-b010-83df9483eb9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:35:38.234709 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234657 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-workload-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:35:38.234709 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234675 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-credential-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:35:38.235057 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234731 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51091562-0995-47b8-b010-83df9483eb9e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51091562-0995-47b8-b010-83df9483eb9e" (UID: "51091562-0995-47b8-b010-83df9483eb9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:35:38.235057 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.234904 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51091562-0995-47b8-b010-83df9483eb9e" (UID: "51091562-0995-47b8-b010-83df9483eb9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:35:38.235270 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.235242 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-istio-data" (OuterVolumeSpecName: "istio-data") pod "51091562-0995-47b8-b010-83df9483eb9e" (UID: "51091562-0995-47b8-b010-83df9483eb9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:35:38.236530 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.236483 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51091562-0995-47b8-b010-83df9483eb9e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51091562-0995-47b8-b010-83df9483eb9e" (UID: "51091562-0995-47b8-b010-83df9483eb9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 20:35:38.236620 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.236540 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51091562-0995-47b8-b010-83df9483eb9e-istio-token" (OuterVolumeSpecName: "istio-token") pod "51091562-0995-47b8-b010-83df9483eb9e" (UID: "51091562-0995-47b8-b010-83df9483eb9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:35:38.236620 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.236573 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51091562-0995-47b8-b010-83df9483eb9e-kube-api-access-fxxts" (OuterVolumeSpecName: "kube-api-access-fxxts") pod "51091562-0995-47b8-b010-83df9483eb9e" (UID: "51091562-0995-47b8-b010-83df9483eb9e"). InnerVolumeSpecName "kube-api-access-fxxts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:35:38.236620 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.236596 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51091562-0995-47b8-b010-83df9483eb9e" (UID: "51091562-0995-47b8-b010-83df9483eb9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:35:38.335014 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.334967 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxxts\" (UniqueName: \"kubernetes.io/projected/51091562-0995-47b8-b010-83df9483eb9e-kube-api-access-fxxts\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:35:38.335014 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.335013 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-workload-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:35:38.335147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.335024 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51091562-0995-47b8-b010-83df9483eb9e-istiod-ca-cert\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:35:38.335147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.335032 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51091562-0995-47b8-b010-83df9483eb9e-istio-token\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:35:38.335147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.335039 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51091562-0995-47b8-b010-83df9483eb9e-istio-podinfo\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:35:38.335147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.335046 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-istio-envoy\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:35:38.335147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.335054 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51091562-0995-47b8-b010-83df9483eb9e-istio-data\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:35:38.602372 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.602271 2575 generic.go:356] "Generic (PLEG): container finished" podID="51091562-0995-47b8-b010-83df9483eb9e" containerID="b9493f7b4c1fd4a48b949225a1ad0e7431eaf15341a76a19cad635e7bac3d48a" exitCode=0 Jul 22 20:35:38.602372 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.602330 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" event={"ID":"51091562-0995-47b8-b010-83df9483eb9e","Type":"ContainerDied","Data":"b9493f7b4c1fd4a48b949225a1ad0e7431eaf15341a76a19cad635e7bac3d48a"} Jul 22 20:35:38.602372 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.602339 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" Jul 22 20:35:38.602628 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.602395 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh" event={"ID":"51091562-0995-47b8-b010-83df9483eb9e","Type":"ContainerDied","Data":"3e87925679d3d75b19ac81b4c51043228110f2ab9c184c1de6f82c00e3e50e61"} Jul 22 20:35:38.602628 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.602416 2575 scope.go:117] "RemoveContainer" containerID="b9493f7b4c1fd4a48b949225a1ad0e7431eaf15341a76a19cad635e7bac3d48a" Jul 22 20:35:38.613339 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.613316 2575 scope.go:117] "RemoveContainer" containerID="f6d23bbb8473dd09821dbb8f94dc108f7481be3f0f93c2fa218221d615b3de70" Jul 22 20:35:38.620620 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.620604 2575 scope.go:117] "RemoveContainer" containerID="b9493f7b4c1fd4a48b949225a1ad0e7431eaf15341a76a19cad635e7bac3d48a" Jul 22 20:35:38.620870 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:35:38.620851 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9493f7b4c1fd4a48b949225a1ad0e7431eaf15341a76a19cad635e7bac3d48a\": container with ID starting with b9493f7b4c1fd4a48b949225a1ad0e7431eaf15341a76a19cad635e7bac3d48a not found: ID does not exist" containerID="b9493f7b4c1fd4a48b949225a1ad0e7431eaf15341a76a19cad635e7bac3d48a" Jul 22 20:35:38.620940 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.620878 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9493f7b4c1fd4a48b949225a1ad0e7431eaf15341a76a19cad635e7bac3d48a"} err="failed to get container status \"b9493f7b4c1fd4a48b949225a1ad0e7431eaf15341a76a19cad635e7bac3d48a\": rpc error: code = NotFound desc = could not find container \"b9493f7b4c1fd4a48b949225a1ad0e7431eaf15341a76a19cad635e7bac3d48a\": container with ID starting with b9493f7b4c1fd4a48b949225a1ad0e7431eaf15341a76a19cad635e7bac3d48a not found: ID does not exist" Jul 22 20:35:38.620940 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.620895 2575 scope.go:117] "RemoveContainer" containerID="f6d23bbb8473dd09821dbb8f94dc108f7481be3f0f93c2fa218221d615b3de70" Jul 22 20:35:38.621190 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:35:38.621156 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d23bbb8473dd09821dbb8f94dc108f7481be3f0f93c2fa218221d615b3de70\": container with ID starting with f6d23bbb8473dd09821dbb8f94dc108f7481be3f0f93c2fa218221d615b3de70 not found: ID does not exist" containerID="f6d23bbb8473dd09821dbb8f94dc108f7481be3f0f93c2fa218221d615b3de70" Jul 22 20:35:38.621262 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.621195 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d23bbb8473dd09821dbb8f94dc108f7481be3f0f93c2fa218221d615b3de70"} err="failed to get container status \"f6d23bbb8473dd09821dbb8f94dc108f7481be3f0f93c2fa218221d615b3de70\": rpc error: code = NotFound desc = could not find container \"f6d23bbb8473dd09821dbb8f94dc108f7481be3f0f93c2fa218221d615b3de70\": container with ID starting with f6d23bbb8473dd09821dbb8f94dc108f7481be3f0f93c2fa218221d615b3de70 not found: ID does not exist" Jul 22 20:35:38.623844 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.623820 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh"] Jul 22 20:35:38.627661 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:38.627633 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6627cefd-openshift-default-8565f58855-8gldh"] Jul 22 20:35:40.006665 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:40.006633 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51091562-0995-47b8-b010-83df9483eb9e" path="/var/lib/kubelet/pods/51091562-0995-47b8-b010-83df9483eb9e/volumes" Jul 22 20:35:41.059476 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:41.059437 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:41.169092 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:41.169058 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:41.259996 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:41.259945 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:41.361415 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:41.361320 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:35:41.710994 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:35:41.710954 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:07.721198 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:07.721101 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:14.100389 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:14.100340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:14.202010 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:14.201949 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:14.298646 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:14.298611 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:30.193274 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:30.193238 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:35.889049 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:35.889009 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:35.952886 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:35.952831 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:36.043487 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:36.043446 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:40.824880 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:40.824848 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:36:40.826667 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:40.826650 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:36:43.681351 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:43.681317 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:46.614532 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:46.614497 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:47.400459 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:47.400423 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:53.331187 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:53.331151 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:53.428136 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:53.428094 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:53.528637 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:53.528603 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:53.628841 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:53.628759 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:54.404756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:54.404721 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:54.504492 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:54.504459 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:54.607345 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:54.607298 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:56.211959 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:56.211927 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:36:57.971030 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:36:57.970996 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:37:02.557586 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:02.557537 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:02.868921 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:02.868830 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:03.247284 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:03.247255 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:07.674049 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:07.674010 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:37:07.702265 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:07.702227 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:09.129543 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.129509 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:09.226942 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.226908 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:09.366591 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.366558 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:09.652641 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.652609 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5"] Jul 22 20:37:09.652872 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.652862 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" containerName="istio-proxy" Jul 22 20:37:09.652914 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.652874 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" containerName="istio-proxy" Jul 22 20:37:09.652914 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.652893 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51091562-0995-47b8-b010-83df9483eb9e" containerName="istio-proxy" Jul 22 20:37:09.652914 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.652900 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="51091562-0995-47b8-b010-83df9483eb9e" containerName="istio-proxy" Jul 22 20:37:09.652914 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.652906 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51091562-0995-47b8-b010-83df9483eb9e" containerName="istio-proxy" Jul 22 20:37:09.652914 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.652911 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="51091562-0995-47b8-b010-83df9483eb9e" containerName="istio-proxy" Jul 22 20:37:09.653079 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.652918 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c740631-a06b-472c-bc75-2e9920a71125" containerName="revoke-keys" Jul 22 20:37:09.653079 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.652924 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c740631-a06b-472c-bc75-2e9920a71125" containerName="revoke-keys" Jul 22 20:37:09.653079 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.652969 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf5e66b-5797-4ae1-bd39-8a7a286b5727" containerName="istio-proxy" Jul 22 20:37:09.653079 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.653001 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="51091562-0995-47b8-b010-83df9483eb9e" containerName="istio-proxy" Jul 22 20:37:09.653079 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.653006 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c740631-a06b-472c-bc75-2e9920a71125" containerName="revoke-keys" Jul 22 20:37:09.653079 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.653011 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="51091562-0995-47b8-b010-83df9483eb9e" containerName="istio-proxy" Jul 22 20:37:09.656159 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.656142 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.658432 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.658407 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c6819716-openshift-default-dockercfg-9mx87\"" Jul 22 20:37:09.672768 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.672744 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:09.677062 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.677041 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5"] Jul 22 20:37:09.772961 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.772922 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-workload-socket\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.772961 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.772972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-credential-socket\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.773197 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.773010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-podinfo\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.773197 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.773087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-data\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.773197 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.773133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdnpb\" (UniqueName: \"kubernetes.io/projected/9c103010-01f4-4c16-8de3-194f40cef5d4-kube-api-access-wdnpb\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.773197 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.773154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-envoy\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.773351 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.773200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c103010-01f4-4c16-8de3-194f40cef5d4-istiod-ca-cert\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.773351 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.773238 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-workload-certs\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.773351 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.773286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-token\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.874198 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.874161 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdnpb\" (UniqueName: \"kubernetes.io/projected/9c103010-01f4-4c16-8de3-194f40cef5d4-kube-api-access-wdnpb\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.874198 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.874200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-envoy\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.874464 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.874225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c103010-01f4-4c16-8de3-194f40cef5d4-istiod-ca-cert\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.874464 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.874267 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-workload-certs\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.874464 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.874299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-token\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.874464 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.874331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-workload-socket\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.874464 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.874358 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-credential-socket\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.874464 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.874389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-podinfo\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.874464 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.874438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-data\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.874805 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.874769 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-workload-certs\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.874868 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.874796 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-workload-socket\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.874868 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.874851 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-credential-socket\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.874970 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.874905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-data\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.875213 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.875195 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c103010-01f4-4c16-8de3-194f40cef5d4-istiod-ca-cert\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.876556 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.876534 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-envoy\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.876752 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.876734 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-podinfo\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.888675 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.888647 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-token\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.888810 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.888742 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdnpb\" (UniqueName: \"kubernetes.io/projected/9c103010-01f4-4c16-8de3-194f40cef5d4-kube-api-access-wdnpb\") pod \"e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:09.959766 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.959728 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:09.965215 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:09.965190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:10.099992 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:10.099952 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5"] Jul 22 20:37:10.102036 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:37:10.102008 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c103010_01f4_4c16_8de3_194f40cef5d4.slice/crio-1e8696b509a5ec7a6b750f11e92c577790986ea78e8b72ff60c0476f197d19e8 WatchSource:0}: Error finding container 1e8696b509a5ec7a6b750f11e92c577790986ea78e8b72ff60c0476f197d19e8: Status 404 returned error can't find the container with id 1e8696b509a5ec7a6b750f11e92c577790986ea78e8b72ff60c0476f197d19e8 Jul 22 20:37:10.104163 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:10.104124 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:37:10.104229 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:10.104203 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:37:10.104264 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:10.104238 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:37:10.980051 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:10.980010 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" event={"ID":"9c103010-01f4-4c16-8de3-194f40cef5d4","Type":"ContainerStarted","Data":"297fe91ba265d2d5fdedd4462864e9f655344305cc8d556651fd7588201fe4d4"} Jul 22 20:37:10.980448 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:10.980096 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" event={"ID":"9c103010-01f4-4c16-8de3-194f40cef5d4","Type":"ContainerStarted","Data":"1e8696b509a5ec7a6b750f11e92c577790986ea78e8b72ff60c0476f197d19e8"} Jul 22 20:37:11.003357 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:11.003308 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" podStartSLOduration=2.003294661 podStartE2EDuration="2.003294661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:37:11.002458107 +0000 UTC m=+2131.515337815" watchObservedRunningTime="2026-07-22 20:37:11.003294661 +0000 UTC m=+2131.516174366" Jul 22 20:37:11.965955 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:11.965912 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:11.970864 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:11.970838 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:11.985403 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:11.985373 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:11.986390 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:11.986369 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:12.934008 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:12.933956 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:13.020812 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:13.020778 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:13.083925 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:13.083886 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:13.268753 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:13.268717 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:13.702907 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:13.702869 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:13.828669 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:13.828628 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:17.471751 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:17.471716 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:17.873005 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:17.872901 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:37:18.142639 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:18.142543 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:18.242549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:18.242511 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:18.589371 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:18.589340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:24.325876 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:24.325839 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:24.426051 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:24.426013 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:24.694836 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:24.694802 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm"] Jul 22 20:37:24.697737 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:24.697717 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm" Jul 22 20:37:24.700247 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:24.700211 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjqt7\"" Jul 22 20:37:24.700405 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:24.700333 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 20:37:24.706494 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:24.705746 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm"] Jul 22 20:37:24.808108 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:24.808069 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cce0861-6d9f-4715-ad0e-a8d108d77adb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6819716-mv7wm\" (UID: \"6cce0861-6d9f-4715-ad0e-a8d108d77adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm" Jul 22 20:37:24.909316 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:24.909274 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cce0861-6d9f-4715-ad0e-a8d108d77adb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6819716-mv7wm\" (UID: \"6cce0861-6d9f-4715-ad0e-a8d108d77adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm" Jul 22 20:37:24.909932 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:24.909910 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cce0861-6d9f-4715-ad0e-a8d108d77adb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6819716-mv7wm\" (UID: \"6cce0861-6d9f-4715-ad0e-a8d108d77adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm" Jul 22 20:37:25.011450 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:25.011368 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm" Jul 22 20:37:25.157672 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:25.157642 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm"] Jul 22 20:37:25.160301 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:37:25.160262 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cce0861_6d9f_4715_ad0e_a8d108d77adb.slice/crio-1bdb6816b0abca202a39407af698bef0c87e7c720b9bad4ad5153c3fa465c45d WatchSource:0}: Error finding container 1bdb6816b0abca202a39407af698bef0c87e7c720b9bad4ad5153c3fa465c45d: Status 404 returned error can't find the container with id 1bdb6816b0abca202a39407af698bef0c87e7c720b9bad4ad5153c3fa465c45d Jul 22 20:37:26.049170 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:26.049139 2575 generic.go:356] "Generic (PLEG): container finished" podID="6cce0861-6d9f-4715-ad0e-a8d108d77adb" containerID="742df2d86700963634969895afe7c453db634e6df3e43658d3c3604af60e0c27" exitCode=0 Jul 22 20:37:26.049628 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:26.049223 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm" event={"ID":"6cce0861-6d9f-4715-ad0e-a8d108d77adb","Type":"ContainerDied","Data":"742df2d86700963634969895afe7c453db634e6df3e43658d3c3604af60e0c27"} Jul 22 20:37:26.049628 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:26.049279 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm" event={"ID":"6cce0861-6d9f-4715-ad0e-a8d108d77adb","Type":"ContainerStarted","Data":"1bdb6816b0abca202a39407af698bef0c87e7c720b9bad4ad5153c3fa465c45d"} Jul 22 20:37:26.050838 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:26.050816 2575 generic.go:356] "Generic (PLEG): container finished" podID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerID="297fe91ba265d2d5fdedd4462864e9f655344305cc8d556651fd7588201fe4d4" exitCode=0 Jul 22 20:37:26.050942 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:26.050853 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" event={"ID":"9c103010-01f4-4c16-8de3-194f40cef5d4","Type":"ContainerDied","Data":"297fe91ba265d2d5fdedd4462864e9f655344305cc8d556651fd7588201fe4d4"} Jul 22 20:37:26.051234 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:26.051222 2575 scope.go:117] "RemoveContainer" containerID="297fe91ba265d2d5fdedd4462864e9f655344305cc8d556651fd7588201fe4d4" Jul 22 20:37:26.052177 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:26.052151 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:37:26.052254 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:26.052217 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:37:26.052254 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:26.052250 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:37:26.985649 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:26.985611 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:27.056984 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:27.056933 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" event={"ID":"9c103010-01f4-4c16-8de3-194f40cef5d4","Type":"ContainerStarted","Data":"220de85b23b2765d46e139cbd5c763e1d8ac05ce9ef18ad677b7ae2f2b3d8c7a"} Jul 22 20:37:27.224539 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:27.224511 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm" Jul 22 20:37:27.328440 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:27.328348 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cce0861-6d9f-4715-ad0e-a8d108d77adb-maas-api-service-ca\") pod \"6cce0861-6d9f-4715-ad0e-a8d108d77adb\" (UID: \"6cce0861-6d9f-4715-ad0e-a8d108d77adb\") " Jul 22 20:37:27.328698 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:27.328672 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cce0861-6d9f-4715-ad0e-a8d108d77adb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6cce0861-6d9f-4715-ad0e-a8d108d77adb" (UID: "6cce0861-6d9f-4715-ad0e-a8d108d77adb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:37:27.429750 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:27.429703 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cce0861-6d9f-4715-ad0e-a8d108d77adb-maas-api-service-ca\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:37:27.966079 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:27.966038 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:27.967169 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:27.967143 2575 patch_prober.go:28] interesting pod/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 22 20:37:27.967285 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:27.967206 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" podUID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 22 20:37:28.062862 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:28.062825 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm" event={"ID":"6cce0861-6d9f-4715-ad0e-a8d108d77adb","Type":"ContainerDied","Data":"1bdb6816b0abca202a39407af698bef0c87e7c720b9bad4ad5153c3fa465c45d"} Jul 22 20:37:28.062862 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:28.062849 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm" Jul 22 20:37:28.063385 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:28.062878 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bdb6816b0abca202a39407af698bef0c87e7c720b9bad4ad5153c3fa465c45d" Jul 22 20:37:28.965959 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:28.965919 2575 patch_prober.go:28] interesting pod/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 22 20:37:28.966160 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:28.966003 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" podUID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 22 20:37:29.966090 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:29.966056 2575 patch_prober.go:28] interesting pod/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 22 20:37:29.966472 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:29.966109 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" podUID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 22 20:37:30.965633 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:30.965586 2575 patch_prober.go:28] interesting pod/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 22 20:37:30.965831 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:30.965648 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" podUID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 22 20:37:31.970029 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:31.969970 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:31.970399 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:31.970365 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:31.971057 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:31.971037 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:35.266649 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:35.266609 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:38.163994 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:38.163942 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:37:39.767143 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:39.767101 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm"] Jul 22 20:37:39.770577 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:39.770536 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6819716-mv7wm"] Jul 22 20:37:39.770577 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:39.770553 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:40.006350 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:40.006314 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cce0861-6d9f-4715-ad0e-a8d108d77adb" path="/var/lib/kubelet/pods/6cce0861-6d9f-4715-ad0e-a8d108d77adb/volumes" Jul 22 20:37:40.168145 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:40.168053 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:45.785190 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:45.785150 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:45.851652 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:45.851611 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:46.035042 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:46.035011 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5"] Jul 22 20:37:46.035314 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:46.035257 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" podUID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerName="istio-proxy" containerID="cri-o://220de85b23b2765d46e139cbd5c763e1d8ac05ce9ef18ad677b7ae2f2b3d8c7a" gracePeriod=30 Jul 22 20:37:46.059190 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:46.059143 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:46.304384 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:46.304291 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:46.427063 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:46.427013 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:46.521762 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:46.521727 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:46.653425 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:46.653341 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:46.972851 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:46.972815 2575 patch_prober.go:28] interesting pod/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 20:37:46.973241 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:46.972875 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" podUID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 20:37:51.166716 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.166683 2575 generic.go:356] "Generic (PLEG): container finished" podID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerID="220de85b23b2765d46e139cbd5c763e1d8ac05ce9ef18ad677b7ae2f2b3d8c7a" exitCode=0 Jul 22 20:37:51.167075 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.166765 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" event={"ID":"9c103010-01f4-4c16-8de3-194f40cef5d4","Type":"ContainerDied","Data":"220de85b23b2765d46e139cbd5c763e1d8ac05ce9ef18ad677b7ae2f2b3d8c7a"} Jul 22 20:37:51.167075 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.166854 2575 scope.go:117] "RemoveContainer" containerID="297fe91ba265d2d5fdedd4462864e9f655344305cc8d556651fd7588201fe4d4" Jul 22 20:37:51.275427 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.275401 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:51.321654 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.321571 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-credential-socket\") pod \"9c103010-01f4-4c16-8de3-194f40cef5d4\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " Jul 22 20:37:51.321654 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.321605 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-data\") pod \"9c103010-01f4-4c16-8de3-194f40cef5d4\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " Jul 22 20:37:51.321654 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.321622 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-envoy\") pod \"9c103010-01f4-4c16-8de3-194f40cef5d4\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " Jul 22 20:37:51.321654 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.321641 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-workload-socket\") pod \"9c103010-01f4-4c16-8de3-194f40cef5d4\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " Jul 22 20:37:51.321654 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.321658 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-token\") pod \"9c103010-01f4-4c16-8de3-194f40cef5d4\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " Jul 22 20:37:51.322059 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.321697 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-workload-certs\") pod \"9c103010-01f4-4c16-8de3-194f40cef5d4\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " Jul 22 20:37:51.322059 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.321751 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdnpb\" (UniqueName: \"kubernetes.io/projected/9c103010-01f4-4c16-8de3-194f40cef5d4-kube-api-access-wdnpb\") pod \"9c103010-01f4-4c16-8de3-194f40cef5d4\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " Jul 22 20:37:51.322059 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.321799 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-podinfo\") pod \"9c103010-01f4-4c16-8de3-194f40cef5d4\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " Jul 22 20:37:51.322059 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.321855 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c103010-01f4-4c16-8de3-194f40cef5d4-istiod-ca-cert\") pod \"9c103010-01f4-4c16-8de3-194f40cef5d4\" (UID: \"9c103010-01f4-4c16-8de3-194f40cef5d4\") " Jul 22 20:37:51.322059 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.322023 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c103010-01f4-4c16-8de3-194f40cef5d4" (UID: "9c103010-01f4-4c16-8de3-194f40cef5d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:37:51.322322 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.322146 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c103010-01f4-4c16-8de3-194f40cef5d4" (UID: "9c103010-01f4-4c16-8de3-194f40cef5d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:37:51.322322 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.322168 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-credential-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:37:51.322322 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.322211 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c103010-01f4-4c16-8de3-194f40cef5d4" (UID: "9c103010-01f4-4c16-8de3-194f40cef5d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:37:51.322479 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.322327 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c103010-01f4-4c16-8de3-194f40cef5d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c103010-01f4-4c16-8de3-194f40cef5d4" (UID: "9c103010-01f4-4c16-8de3-194f40cef5d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:37:51.322583 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.322563 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c103010-01f4-4c16-8de3-194f40cef5d4" (UID: "9c103010-01f4-4c16-8de3-194f40cef5d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:37:51.323909 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.323885 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c103010-01f4-4c16-8de3-194f40cef5d4" (UID: "9c103010-01f4-4c16-8de3-194f40cef5d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:37:51.324052 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.323912 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c103010-01f4-4c16-8de3-194f40cef5d4-kube-api-access-wdnpb" (OuterVolumeSpecName: "kube-api-access-wdnpb") pod "9c103010-01f4-4c16-8de3-194f40cef5d4" (UID: "9c103010-01f4-4c16-8de3-194f40cef5d4"). InnerVolumeSpecName "kube-api-access-wdnpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:37:51.324196 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.324179 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c103010-01f4-4c16-8de3-194f40cef5d4" (UID: "9c103010-01f4-4c16-8de3-194f40cef5d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:37:51.324294 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.324276 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c103010-01f4-4c16-8de3-194f40cef5d4" (UID: "9c103010-01f4-4c16-8de3-194f40cef5d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 20:37:51.423067 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.423018 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c103010-01f4-4c16-8de3-194f40cef5d4-istiod-ca-cert\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:37:51.423067 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.423061 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-data\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:37:51.423067 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.423070 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-envoy\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:37:51.423067 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.423080 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-workload-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:37:51.423319 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.423088 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-token\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:37:51.423319 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.423096 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c103010-01f4-4c16-8de3-194f40cef5d4-workload-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:37:51.423319 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.423105 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdnpb\" (UniqueName: \"kubernetes.io/projected/9c103010-01f4-4c16-8de3-194f40cef5d4-kube-api-access-wdnpb\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:37:51.423319 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:51.423114 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c103010-01f4-4c16-8de3-194f40cef5d4-istio-podinfo\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:37:52.172350 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:52.172321 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" Jul 22 20:37:52.172813 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:52.172327 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5" event={"ID":"9c103010-01f4-4c16-8de3-194f40cef5d4","Type":"ContainerDied","Data":"1e8696b509a5ec7a6b750f11e92c577790986ea78e8b72ff60c0476f197d19e8"} Jul 22 20:37:52.172813 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:52.172463 2575 scope.go:117] "RemoveContainer" containerID="220de85b23b2765d46e139cbd5c763e1d8ac05ce9ef18ad677b7ae2f2b3d8c7a" Jul 22 20:37:52.189230 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:52.189206 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5"] Jul 22 20:37:52.195604 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:52.195581 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c6819716-openshift-default-8f58c4df5-4m2t5"] Jul 22 20:37:52.200366 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:52.200343 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:52.259508 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:52.259458 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:52.348368 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:52.348277 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:52.452363 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:52.452328 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:52.582368 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:52.582323 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:54.006542 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:54.006513 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c103010-01f4-4c16-8de3-194f40cef5d4" path="/var/lib/kubelet/pods/9c103010-01f4-4c16-8de3-194f40cef5d4/volumes" Jul 22 20:37:58.483834 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:58.483759 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:37:58.584220 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:37:58.584186 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:38:05.588514 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:05.588475 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:38:05.922076 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:05.921956 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x"] Jul 22 20:38:05.922399 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:05.922382 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cce0861-6d9f-4715-ad0e-a8d108d77adb" containerName="revoke-keys" Jul 22 20:38:05.922443 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:05.922402 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cce0861-6d9f-4715-ad0e-a8d108d77adb" containerName="revoke-keys" Jul 22 20:38:05.922443 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:05.922419 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerName="istio-proxy" Jul 22 20:38:05.922443 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:05.922428 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerName="istio-proxy" Jul 22 20:38:05.922538 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:05.922508 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerName="istio-proxy" Jul 22 20:38:05.922538 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:05.922519 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerName="istio-proxy" Jul 22 20:38:05.922538 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:05.922529 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cce0861-6d9f-4715-ad0e-a8d108d77adb" containerName="revoke-keys" Jul 22 20:38:05.925478 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:05.925456 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:05.927795 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:05.927769 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e950683d-openshift-default-dockercfg-7qx54\"" Jul 22 20:38:05.936918 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:05.936889 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x"] Jul 22 20:38:05.946644 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:05.946595 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:38:06.053581 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.053536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-credential-socket\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.053581 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.053584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-envoy\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.053811 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.053607 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istiod-ca-cert\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.053811 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.053670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-podinfo\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.053811 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.053720 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-workload-certs\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.053811 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.053735 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-token\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.053811 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.053757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86lpz\" (UniqueName: \"kubernetes.io/projected/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-kube-api-access-86lpz\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.054005 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.053852 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-data\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.054005 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.053887 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-workload-socket\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.155288 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.155246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-envoy\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.155288 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.155292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istiod-ca-cert\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.155499 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.155440 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-podinfo\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.155539 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.155521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-workload-certs\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.155591 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.155548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-token\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.155767 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.155739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86lpz\" (UniqueName: \"kubernetes.io/projected/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-kube-api-access-86lpz\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.155873 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.155855 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-workload-certs\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.155996 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.155958 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-data\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.156071 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.156033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-workload-socket\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.156129 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.156099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-credential-socket\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.156129 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.156111 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istiod-ca-cert\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.156379 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.156357 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-workload-socket\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.156457 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.156434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-credential-socket\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.156612 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.156593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-data\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.157858 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.157821 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-envoy\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.158101 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.158084 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-podinfo\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.166617 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.166586 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-token\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.167233 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.167215 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86lpz\" (UniqueName: \"kubernetes.io/projected/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-kube-api-access-86lpz\") pod \"e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.236491 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.236451 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:06.375934 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.375903 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x"] Jul 22 20:38:06.378853 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:38:06.378825 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14386fc5_73e6_4a13_befa_a3a1b50b0fe2.slice/crio-43f3156f883469dc528c6459ce378d98e06f306ffbec675d2a1e4593690e78d0 WatchSource:0}: Error finding container 43f3156f883469dc528c6459ce378d98e06f306ffbec675d2a1e4593690e78d0: Status 404 returned error can't find the container with id 43f3156f883469dc528c6459ce378d98e06f306ffbec675d2a1e4593690e78d0 Jul 22 20:38:06.380950 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.380903 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:38:06.381067 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.381015 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:38:06.381067 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.381048 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:38:06.435688 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:06.435646 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:38:07.238586 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:07.238545 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" event={"ID":"14386fc5-73e6-4a13-befa-a3a1b50b0fe2","Type":"ContainerStarted","Data":"c7bc2fb683e772500e8c3436b133acdad6a3a0a123ebbe620c338fdd645e29c4"} Jul 22 20:38:07.238957 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:07.238606 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" event={"ID":"14386fc5-73e6-4a13-befa-a3a1b50b0fe2","Type":"ContainerStarted","Data":"43f3156f883469dc528c6459ce378d98e06f306ffbec675d2a1e4593690e78d0"} Jul 22 20:38:07.257755 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:07.257690 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" podStartSLOduration=2.2576710159999998 podStartE2EDuration="2.257671016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:38:07.257076579 +0000 UTC m=+2187.769956287" watchObservedRunningTime="2026-07-22 20:38:07.257671016 +0000 UTC m=+2187.770550932" Jul 22 20:38:07.886257 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:07.886226 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:38:08.237056 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:08.237022 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:08.242687 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:08.242660 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:09.212192 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:09.212142 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:38:09.247751 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:09.247716 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:09.248872 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:09.248848 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:09.311760 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:09.311724 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:38:09.415438 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:09.415397 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:38:09.582842 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:09.582746 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:38:09.880510 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:09.880411 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:38:09.984482 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:09.984452 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:38:13.773281 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:13.773245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:38:14.568632 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:14.568596 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:38:19.333250 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.333205 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg"] Jul 22 20:38:19.333711 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.333690 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerName="istio-proxy" Jul 22 20:38:19.333755 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.333715 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c103010-01f4-4c16-8de3-194f40cef5d4" containerName="istio-proxy" Jul 22 20:38:19.335905 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.335885 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.339138 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.339107 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e950683d\"/\"default-dockercfg-7xnjw\"" Jul 22 20:38:19.339386 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.339144 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e950683d\"/\"kube-root-ca.crt\"" Jul 22 20:38:19.339468 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.339260 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e950683d\"/\"openshift-service-ca.crt\"" Jul 22 20:38:19.339521 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.339265 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e950683d\"/\"e2e-lifecycle-model-e950683d-kserve-self-signed-certs\"" Jul 22 20:38:19.345415 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.345065 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg"] Jul 22 20:38:19.366080 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.366043 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:38:19.474547 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.474504 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-dshm\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.474722 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.474553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-tmp-dir\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.474722 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.474688 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-home\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.474812 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.474792 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f498\" (UniqueName: \"kubernetes.io/projected/6adb8c22-1296-4d63-b71b-1b2061d97406-kube-api-access-4f498\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.474853 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.474823 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-model-cache\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.474853 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.474847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6adb8c22-1296-4d63-b71b-1b2061d97406-tls-certs\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.575897 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.575860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-home\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.576114 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.575927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f498\" (UniqueName: \"kubernetes.io/projected/6adb8c22-1296-4d63-b71b-1b2061d97406-kube-api-access-4f498\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.576114 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.575946 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-model-cache\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.576114 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.575962 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6adb8c22-1296-4d63-b71b-1b2061d97406-tls-certs\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.576114 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.576008 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-dshm\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.576114 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.576090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-tmp-dir\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.576686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.576365 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-home\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.576686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.576420 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-model-cache\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.576686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.576522 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-tmp-dir\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.578431 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.578408 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-dshm\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.578578 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.578558 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6adb8c22-1296-4d63-b71b-1b2061d97406-tls-certs\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.584146 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.584081 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f498\" (UniqueName: \"kubernetes.io/projected/6adb8c22-1296-4d63-b71b-1b2061d97406-kube-api-access-4f498\") pod \"e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.647759 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.647724 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:19.791148 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:19.791111 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg"] Jul 22 20:38:19.794170 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:38:19.794132 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6adb8c22_1296_4d63_b71b_1b2061d97406.slice/crio-1c68c669156de64feec247e4253e1c47bf7f58c441e2eee9435f223c9b4a7f34 WatchSource:0}: Error finding container 1c68c669156de64feec247e4253e1c47bf7f58c441e2eee9435f223c9b4a7f34: Status 404 returned error can't find the container with id 1c68c669156de64feec247e4253e1c47bf7f58c441e2eee9435f223c9b4a7f34 Jul 22 20:38:20.294798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:20.294754 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" event={"ID":"6adb8c22-1296-4d63-b71b-1b2061d97406","Type":"ContainerStarted","Data":"d66c7e79f0d1381f5987e8f27609a9fc6b348bd8bf13134e9d1bdf37d754cef7"} Jul 22 20:38:20.294990 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:20.294811 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" event={"ID":"6adb8c22-1296-4d63-b71b-1b2061d97406","Type":"ContainerStarted","Data":"1c68c669156de64feec247e4253e1c47bf7f58c441e2eee9435f223c9b4a7f34"} Jul 22 20:38:20.311050 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:20.310967 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" podStartSLOduration=1.3109423310000001 podStartE2EDuration="1.310942331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:38:20.310500297 +0000 UTC m=+2200.823380034" watchObservedRunningTime="2026-07-22 20:38:20.310942331 +0000 UTC m=+2200.823822038" Jul 22 20:38:22.304952 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:22.304856 2575 generic.go:356] "Generic (PLEG): container finished" podID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" containerID="c7bc2fb683e772500e8c3436b133acdad6a3a0a123ebbe620c338fdd645e29c4" exitCode=0 Jul 22 20:38:22.304952 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:22.304912 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" event={"ID":"14386fc5-73e6-4a13-befa-a3a1b50b0fe2","Type":"ContainerDied","Data":"c7bc2fb683e772500e8c3436b133acdad6a3a0a123ebbe620c338fdd645e29c4"} Jul 22 20:38:22.305444 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:22.305299 2575 scope.go:117] "RemoveContainer" containerID="c7bc2fb683e772500e8c3436b133acdad6a3a0a123ebbe620c338fdd645e29c4" Jul 22 20:38:22.306340 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:22.306299 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:38:22.306479 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:22.306388 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:38:22.306479 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:22.306435 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:38:22.874709 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:22.874668 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:38:23.315391 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:23.315345 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" event={"ID":"14386fc5-73e6-4a13-befa-a3a1b50b0fe2","Type":"ContainerStarted","Data":"773aacd0a2ec81fe930d79a161d14e4930764980021bc8424ac2dbc535e876eb"} Jul 22 20:38:24.237426 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:24.237382 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:24.238575 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:24.238547 2575 patch_prober.go:28] interesting pod/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 22 20:38:24.238690 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:24.238598 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" podUID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 22 20:38:24.247831 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:24.247807 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:25.237237 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:25.237193 2575 patch_prober.go:28] interesting pod/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 22 20:38:25.237647 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:25.237252 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" podUID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 22 20:38:26.237046 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:26.237002 2575 patch_prober.go:28] interesting pod/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 22 20:38:26.237264 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:26.237063 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" podUID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 22 20:38:27.296261 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:27.296225 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:27.297381 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:27.297357 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:38:29.648467 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:29.648428 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:29.648940 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:29.648481 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:29.660802 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:29.660772 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:30.364840 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:30.364812 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:38:38.570262 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:38.570221 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:38:56.072798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:56.072756 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:38:57.676589 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:38:57.676551 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:02.350247 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:02.350139 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:04.862799 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:04.862753 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:06.572893 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:06.572849 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:39:09.724929 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:09.724880 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg"] Jul 22 20:39:09.784264 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:09.784226 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:24.745067 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:24.745020 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" podUID="6adb8c22-1296-4d63-b71b-1b2061d97406" containerName="main" containerID="cri-o://d66c7e79f0d1381f5987e8f27609a9fc6b348bd8bf13134e9d1bdf37d754cef7" gracePeriod=45 Jul 22 20:39:24.989304 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:24.989280 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:39:25.044468 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.044378 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-tmp-dir\") pod \"6adb8c22-1296-4d63-b71b-1b2061d97406\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " Jul 22 20:39:25.044468 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.044426 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-model-cache\") pod \"6adb8c22-1296-4d63-b71b-1b2061d97406\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " Jul 22 20:39:25.044468 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.044444 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6adb8c22-1296-4d63-b71b-1b2061d97406-tls-certs\") pod \"6adb8c22-1296-4d63-b71b-1b2061d97406\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " Jul 22 20:39:25.044468 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.044467 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-dshm\") pod \"6adb8c22-1296-4d63-b71b-1b2061d97406\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " Jul 22 20:39:25.044787 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.044488 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f498\" (UniqueName: \"kubernetes.io/projected/6adb8c22-1296-4d63-b71b-1b2061d97406-kube-api-access-4f498\") pod \"6adb8c22-1296-4d63-b71b-1b2061d97406\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " Jul 22 20:39:25.044787 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.044533 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-home\") pod \"6adb8c22-1296-4d63-b71b-1b2061d97406\" (UID: \"6adb8c22-1296-4d63-b71b-1b2061d97406\") " Jul 22 20:39:25.044787 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.044708 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6adb8c22-1296-4d63-b71b-1b2061d97406" (UID: "6adb8c22-1296-4d63-b71b-1b2061d97406"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:39:25.044787 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.044726 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-model-cache" (OuterVolumeSpecName: "model-cache") pod "6adb8c22-1296-4d63-b71b-1b2061d97406" (UID: "6adb8c22-1296-4d63-b71b-1b2061d97406"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:39:25.044955 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.044897 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-home" (OuterVolumeSpecName: "home") pod "6adb8c22-1296-4d63-b71b-1b2061d97406" (UID: "6adb8c22-1296-4d63-b71b-1b2061d97406"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:39:25.046720 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.046686 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adb8c22-1296-4d63-b71b-1b2061d97406-kube-api-access-4f498" (OuterVolumeSpecName: "kube-api-access-4f498") pod "6adb8c22-1296-4d63-b71b-1b2061d97406" (UID: "6adb8c22-1296-4d63-b71b-1b2061d97406"). InnerVolumeSpecName "kube-api-access-4f498". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:39:25.046720 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.046710 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6adb8c22-1296-4d63-b71b-1b2061d97406-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6adb8c22-1296-4d63-b71b-1b2061d97406" (UID: "6adb8c22-1296-4d63-b71b-1b2061d97406"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 20:39:25.046871 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.046843 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-dshm" (OuterVolumeSpecName: "dshm") pod "6adb8c22-1296-4d63-b71b-1b2061d97406" (UID: "6adb8c22-1296-4d63-b71b-1b2061d97406"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:39:25.150381 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.146266 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-tmp-dir\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:25.150381 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.146335 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-model-cache\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:25.150381 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.146348 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6adb8c22-1296-4d63-b71b-1b2061d97406-tls-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:25.150381 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.146360 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-dshm\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:25.151041 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.150997 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f498\" (UniqueName: \"kubernetes.io/projected/6adb8c22-1296-4d63-b71b-1b2061d97406-kube-api-access-4f498\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:25.151183 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.151172 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6adb8c22-1296-4d63-b71b-1b2061d97406-home\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:25.599301 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.599265 2575 generic.go:356] "Generic (PLEG): container finished" podID="6adb8c22-1296-4d63-b71b-1b2061d97406" containerID="d66c7e79f0d1381f5987e8f27609a9fc6b348bd8bf13134e9d1bdf37d754cef7" exitCode=0 Jul 22 20:39:25.599481 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.599319 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" Jul 22 20:39:25.599481 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.599348 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" event={"ID":"6adb8c22-1296-4d63-b71b-1b2061d97406","Type":"ContainerDied","Data":"d66c7e79f0d1381f5987e8f27609a9fc6b348bd8bf13134e9d1bdf37d754cef7"} Jul 22 20:39:25.599481 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.599393 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg" event={"ID":"6adb8c22-1296-4d63-b71b-1b2061d97406","Type":"ContainerDied","Data":"1c68c669156de64feec247e4253e1c47bf7f58c441e2eee9435f223c9b4a7f34"} Jul 22 20:39:25.599481 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.599411 2575 scope.go:117] "RemoveContainer" containerID="d66c7e79f0d1381f5987e8f27609a9fc6b348bd8bf13134e9d1bdf37d754cef7" Jul 22 20:39:25.607817 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.607799 2575 scope.go:117] "RemoveContainer" containerID="d66c7e79f0d1381f5987e8f27609a9fc6b348bd8bf13134e9d1bdf37d754cef7" Jul 22 20:39:25.608136 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:39:25.608113 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66c7e79f0d1381f5987e8f27609a9fc6b348bd8bf13134e9d1bdf37d754cef7\": container with ID starting with d66c7e79f0d1381f5987e8f27609a9fc6b348bd8bf13134e9d1bdf37d754cef7 not found: ID does not exist" containerID="d66c7e79f0d1381f5987e8f27609a9fc6b348bd8bf13134e9d1bdf37d754cef7" Jul 22 20:39:25.608212 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.608149 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66c7e79f0d1381f5987e8f27609a9fc6b348bd8bf13134e9d1bdf37d754cef7"} err="failed to get container status \"d66c7e79f0d1381f5987e8f27609a9fc6b348bd8bf13134e9d1bdf37d754cef7\": rpc error: code = NotFound desc = could not find container \"d66c7e79f0d1381f5987e8f27609a9fc6b348bd8bf13134e9d1bdf37d754cef7\": container with ID starting with d66c7e79f0d1381f5987e8f27609a9fc6b348bd8bf13134e9d1bdf37d754cef7 not found: ID does not exist" Jul 22 20:39:25.619134 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.619109 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg"] Jul 22 20:39:25.624690 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:25.624669 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e950683d/e2e-lifecycle-model-e950683d-kserve-66b7c4cfc5-6prgg"] Jul 22 20:39:26.006738 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:26.006707 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6adb8c22-1296-4d63-b71b-1b2061d97406" path="/var/lib/kubelet/pods/6adb8c22-1296-4d63-b71b-1b2061d97406/volumes" Jul 22 20:39:40.082561 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:40.082528 2575 scope.go:117] "RemoveContainer" containerID="f0cec09b31cfad83147902e497c022ee197257c5f88a10c82fc27389d7c8cf66" Jul 22 20:39:45.884067 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:45.884031 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:46.991315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:46.991274 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:52.410050 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:52.410006 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:52.510906 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:52.510869 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:52.664788 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:52.664711 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x"] Jul 22 20:39:52.665023 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:52.664963 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" podUID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" containerName="istio-proxy" containerID="cri-o://773aacd0a2ec81fe930d79a161d14e4930764980021bc8424ac2dbc535e876eb" gracePeriod=30 Jul 22 20:39:52.690003 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:52.689953 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:52.925900 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:52.925806 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:54.055034 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:54.055000 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:54.155176 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:54.155141 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:54.256402 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:54.256359 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:55.027456 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:55.027418 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:55.125129 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:55.125095 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:55.226700 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:55.226668 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:56.577240 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:56.577206 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:39:56.661006 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:56.660962 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:39:57.299066 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:57.299032 2575 patch_prober.go:28] interesting pod/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 20:39:57.299235 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:57.299102 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" podUID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 20:39:57.907101 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:57.907075 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:39:58.029383 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.029351 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-podinfo\") pod \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " Jul 22 20:39:58.029383 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.029387 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-workload-socket\") pod \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " Jul 22 20:39:58.029383 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.029414 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-token\") pod \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " Jul 22 20:39:58.029743 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.029444 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-envoy\") pod \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " Jul 22 20:39:58.029743 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.029470 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istiod-ca-cert\") pod \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " Jul 22 20:39:58.029743 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.029484 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-data\") pod \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " Jul 22 20:39:58.029743 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.029528 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-credential-socket\") pod \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " Jul 22 20:39:58.029743 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.029560 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-workload-certs\") pod \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " Jul 22 20:39:58.029743 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.029691 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "14386fc5-73e6-4a13-befa-a3a1b50b0fe2" (UID: "14386fc5-73e6-4a13-befa-a3a1b50b0fe2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:39:58.029743 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.029731 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86lpz\" (UniqueName: \"kubernetes.io/projected/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-kube-api-access-86lpz\") pod \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\" (UID: \"14386fc5-73e6-4a13-befa-a3a1b50b0fe2\") " Jul 22 20:39:58.030150 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.030005 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-workload-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:58.030150 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.029998 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "14386fc5-73e6-4a13-befa-a3a1b50b0fe2" (UID: "14386fc5-73e6-4a13-befa-a3a1b50b0fe2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:39:58.030150 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.030042 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "14386fc5-73e6-4a13-befa-a3a1b50b0fe2" (UID: "14386fc5-73e6-4a13-befa-a3a1b50b0fe2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:39:58.030368 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.030334 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "14386fc5-73e6-4a13-befa-a3a1b50b0fe2" (UID: "14386fc5-73e6-4a13-befa-a3a1b50b0fe2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:39:58.030640 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.030616 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-data" (OuterVolumeSpecName: "istio-data") pod "14386fc5-73e6-4a13-befa-a3a1b50b0fe2" (UID: "14386fc5-73e6-4a13-befa-a3a1b50b0fe2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:39:58.031927 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.031900 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "14386fc5-73e6-4a13-befa-a3a1b50b0fe2" (UID: "14386fc5-73e6-4a13-befa-a3a1b50b0fe2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:39:58.032052 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.031945 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-token" (OuterVolumeSpecName: "istio-token") pod "14386fc5-73e6-4a13-befa-a3a1b50b0fe2" (UID: "14386fc5-73e6-4a13-befa-a3a1b50b0fe2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:39:58.032207 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.032181 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "14386fc5-73e6-4a13-befa-a3a1b50b0fe2" (UID: "14386fc5-73e6-4a13-befa-a3a1b50b0fe2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 20:39:58.032300 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.032213 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-kube-api-access-86lpz" (OuterVolumeSpecName: "kube-api-access-86lpz") pod "14386fc5-73e6-4a13-befa-a3a1b50b0fe2" (UID: "14386fc5-73e6-4a13-befa-a3a1b50b0fe2"). InnerVolumeSpecName "kube-api-access-86lpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:39:58.131187 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.131151 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-token\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:58.131187 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.131181 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-envoy\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:58.131187 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.131190 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istiod-ca-cert\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:58.131187 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.131197 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-data\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:58.131462 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.131206 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-credential-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:58.131462 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.131216 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-workload-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:58.131462 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.131225 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86lpz\" (UniqueName: \"kubernetes.io/projected/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-kube-api-access-86lpz\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:58.131462 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.131233 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14386fc5-73e6-4a13-befa-a3a1b50b0fe2-istio-podinfo\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:39:58.743211 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.743177 2575 generic.go:356] "Generic (PLEG): container finished" podID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" containerID="773aacd0a2ec81fe930d79a161d14e4930764980021bc8424ac2dbc535e876eb" exitCode=0 Jul 22 20:39:58.743388 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.743225 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" Jul 22 20:39:58.743388 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.743262 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" event={"ID":"14386fc5-73e6-4a13-befa-a3a1b50b0fe2","Type":"ContainerDied","Data":"773aacd0a2ec81fe930d79a161d14e4930764980021bc8424ac2dbc535e876eb"} Jul 22 20:39:58.743388 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.743319 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x" event={"ID":"14386fc5-73e6-4a13-befa-a3a1b50b0fe2","Type":"ContainerDied","Data":"43f3156f883469dc528c6459ce378d98e06f306ffbec675d2a1e4593690e78d0"} Jul 22 20:39:58.743388 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.743342 2575 scope.go:117] "RemoveContainer" containerID="773aacd0a2ec81fe930d79a161d14e4930764980021bc8424ac2dbc535e876eb" Jul 22 20:39:58.756528 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.756508 2575 scope.go:117] "RemoveContainer" containerID="c7bc2fb683e772500e8c3436b133acdad6a3a0a123ebbe620c338fdd645e29c4" Jul 22 20:39:58.763924 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.763906 2575 scope.go:117] "RemoveContainer" containerID="773aacd0a2ec81fe930d79a161d14e4930764980021bc8424ac2dbc535e876eb" Jul 22 20:39:58.764219 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:39:58.764200 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773aacd0a2ec81fe930d79a161d14e4930764980021bc8424ac2dbc535e876eb\": container with ID starting with 773aacd0a2ec81fe930d79a161d14e4930764980021bc8424ac2dbc535e876eb not found: ID does not exist" containerID="773aacd0a2ec81fe930d79a161d14e4930764980021bc8424ac2dbc535e876eb" Jul 22 20:39:58.764280 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.764227 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773aacd0a2ec81fe930d79a161d14e4930764980021bc8424ac2dbc535e876eb"} err="failed to get container status \"773aacd0a2ec81fe930d79a161d14e4930764980021bc8424ac2dbc535e876eb\": rpc error: code = NotFound desc = could not find container \"773aacd0a2ec81fe930d79a161d14e4930764980021bc8424ac2dbc535e876eb\": container with ID starting with 773aacd0a2ec81fe930d79a161d14e4930764980021bc8424ac2dbc535e876eb not found: ID does not exist" Jul 22 20:39:58.764280 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.764243 2575 scope.go:117] "RemoveContainer" containerID="c7bc2fb683e772500e8c3436b133acdad6a3a0a123ebbe620c338fdd645e29c4" Jul 22 20:39:58.764508 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:39:58.764491 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7bc2fb683e772500e8c3436b133acdad6a3a0a123ebbe620c338fdd645e29c4\": container with ID starting with c7bc2fb683e772500e8c3436b133acdad6a3a0a123ebbe620c338fdd645e29c4 not found: ID does not exist" containerID="c7bc2fb683e772500e8c3436b133acdad6a3a0a123ebbe620c338fdd645e29c4" Jul 22 20:39:58.764551 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.764514 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7bc2fb683e772500e8c3436b133acdad6a3a0a123ebbe620c338fdd645e29c4"} err="failed to get container status \"c7bc2fb683e772500e8c3436b133acdad6a3a0a123ebbe620c338fdd645e29c4\": rpc error: code = NotFound desc = could not find container \"c7bc2fb683e772500e8c3436b133acdad6a3a0a123ebbe620c338fdd645e29c4\": container with ID starting with c7bc2fb683e772500e8c3436b133acdad6a3a0a123ebbe620c338fdd645e29c4 not found: ID does not exist" Jul 22 20:39:58.770244 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.770223 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x"] Jul 22 20:39:58.773482 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:39:58.773462 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e950683d-openshift-default-5b9cc5969f-2426x"] Jul 22 20:40:00.006883 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:00.006850 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" path="/var/lib/kubelet/pods/14386fc5-73e6-4a13-befa-a3a1b50b0fe2/volumes" Jul 22 20:40:02.407095 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:02.407053 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:02.507442 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:02.507400 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:03.121315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:03.121276 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:06.418408 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:06.418374 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:06.520359 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:06.520324 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:08.832348 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:08.832309 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:08.967754 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:08.967719 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:09.063611 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:09.063576 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:09.086384 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:09.086312 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:40:19.581014 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:19.580959 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:19.920672 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:19.920573 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:40.103480 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:40.103406 2575 scope.go:117] "RemoveContainer" containerID="07b4f758cb0064ab792e8a2ae236fce3c4c3bb8254e35a1e3bd458dc177beb88" Jul 22 20:40:40.112692 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:40.112660 2575 scope.go:117] "RemoveContainer" containerID="0bd60622d9f2df2aece829b9d4837dc37bfdbce94f582d8fb179c72ca509c067" Jul 22 20:40:40.918323 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:40.918279 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:46.640582 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:46.640540 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:46.953186 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:46.953153 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:47.054302 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.054259 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:47.359316 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.359228 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:47.602827 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.602795 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k"] Jul 22 20:40:47.603092 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.603078 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" containerName="istio-proxy" Jul 22 20:40:47.603092 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.603094 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" containerName="istio-proxy" Jul 22 20:40:47.603202 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.603105 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6adb8c22-1296-4d63-b71b-1b2061d97406" containerName="main" Jul 22 20:40:47.603202 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.603110 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6adb8c22-1296-4d63-b71b-1b2061d97406" containerName="main" Jul 22 20:40:47.603202 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.603191 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" containerName="istio-proxy" Jul 22 20:40:47.603202 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.603202 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6adb8c22-1296-4d63-b71b-1b2061d97406" containerName="main" Jul 22 20:40:47.603345 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.603211 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" containerName="istio-proxy" Jul 22 20:40:47.606071 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.606054 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.608301 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.608275 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ba38bd-openshift-default-dockercfg-z96lf\"" Jul 22 20:40:47.618362 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.618331 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k"] Jul 22 20:40:47.628989 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.628950 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:47.743087 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.743050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-podinfo\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.743087 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.743092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff36d44-7691-47e1-81fa-e826a9ce49dd-istiod-ca-cert\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.743549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.743114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nltrd\" (UniqueName: \"kubernetes.io/projected/cff36d44-7691-47e1-81fa-e826a9ce49dd-kube-api-access-nltrd\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.743549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.743176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-token\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.743549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.743213 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-workload-socket\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.743549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.743238 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-envoy\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.743549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.743257 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-credential-socket\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.743549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.743317 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-data\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.743549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.743346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-workload-certs\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.829537 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.829497 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:47.844669 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.844636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-data\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.844825 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.844674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-workload-certs\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.844825 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.844706 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-podinfo\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.844825 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.844739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff36d44-7691-47e1-81fa-e826a9ce49dd-istiod-ca-cert\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.844825 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.844761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nltrd\" (UniqueName: \"kubernetes.io/projected/cff36d44-7691-47e1-81fa-e826a9ce49dd-kube-api-access-nltrd\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.844825 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.844805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-token\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.845128 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.844853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-workload-socket\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.845128 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.844889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-envoy\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.845128 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.844925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-credential-socket\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.845291 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.845131 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-data\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.845291 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.845239 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-workload-socket\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.845397 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.845345 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-credential-socket\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.845397 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.845352 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-workload-certs\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.845510 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.845494 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff36d44-7691-47e1-81fa-e826a9ce49dd-istiod-ca-cert\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.847240 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.847220 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-envoy\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.847346 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.847269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-podinfo\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.852383 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.852359 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-token\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.852550 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.852529 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nltrd\" (UniqueName: \"kubernetes.io/projected/cff36d44-7691-47e1-81fa-e826a9ce49dd-kube-api-access-nltrd\") pod \"e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:47.915187 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:47.915093 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:48.049387 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:48.049353 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k"] Jul 22 20:40:48.053049 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:40:48.053023 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff36d44_7691_47e1_81fa_e826a9ce49dd.slice/crio-00a5b9d67d26f2fedc93a5a9616364a0737d2743e853aa164babe1f730af2c3d WatchSource:0}: Error finding container 00a5b9d67d26f2fedc93a5a9616364a0737d2743e853aa164babe1f730af2c3d: Status 404 returned error can't find the container with id 00a5b9d67d26f2fedc93a5a9616364a0737d2743e853aa164babe1f730af2c3d Jul 22 20:40:48.055115 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:48.055084 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:40:48.055206 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:48.055149 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:40:48.055206 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:48.055178 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:40:48.953490 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:48.953451 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" event={"ID":"cff36d44-7691-47e1-81fa-e826a9ce49dd","Type":"ContainerStarted","Data":"ff5573b354cb84e04b78f127604c204b8d38491f11316b7f8645bfcdf947a4f9"} Jul 22 20:40:48.954023 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:48.953511 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" event={"ID":"cff36d44-7691-47e1-81fa-e826a9ce49dd","Type":"ContainerStarted","Data":"00a5b9d67d26f2fedc93a5a9616364a0737d2743e853aa164babe1f730af2c3d"} Jul 22 20:40:48.974289 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:48.974234 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" podStartSLOduration=1.97421901 podStartE2EDuration="1.97421901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:40:48.973164343 +0000 UTC m=+2349.486044054" watchObservedRunningTime="2026-07-22 20:40:48.97421901 +0000 UTC m=+2349.487098715" Jul 22 20:40:49.915476 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:49.915432 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:49.920891 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:49.920865 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:49.957706 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:49.957675 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:49.958705 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:49.958687 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:40:50.909115 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:50.909068 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:51.003999 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:51.003951 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:51.184819 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:51.184722 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:51.512173 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:51.512140 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:51.754095 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:51.754049 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:51.855354 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:51.855267 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:55.415849 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:55.415810 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:55.759306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:55.759247 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:56.066367 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.066274 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv"] Jul 22 20:40:56.066707 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.066688 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" containerName="istio-proxy" Jul 22 20:40:56.066842 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.066709 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="14386fc5-73e6-4a13-befa-a3a1b50b0fe2" containerName="istio-proxy" Jul 22 20:40:56.070145 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.070125 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.072335 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.072316 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-614db5-openshift-default-dockercfg-dhttd\"" Jul 22 20:40:56.079668 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.079644 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv"] Jul 22 20:40:56.093603 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.093571 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:56.114655 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.114615 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-credential-socket\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.114655 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.114656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36df3073-cda4-425b-b552-02eb2b40a494-istio-token\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.114897 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.114685 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-istio-envoy\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.114897 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.114720 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-istio-data\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.114897 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.114787 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-workload-certs\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.114897 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.114827 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36df3073-cda4-425b-b552-02eb2b40a494-istiod-ca-cert\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.114897 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.114879 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-workload-socket\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.115089 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.114922 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36df3073-cda4-425b-b552-02eb2b40a494-istio-podinfo\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.115089 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.114939 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhlb7\" (UniqueName: \"kubernetes.io/projected/36df3073-cda4-425b-b552-02eb2b40a494-kube-api-access-fhlb7\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.215462 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.215421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-workload-socket\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.215462 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.215472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36df3073-cda4-425b-b552-02eb2b40a494-istio-podinfo\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.215723 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.215489 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhlb7\" (UniqueName: \"kubernetes.io/projected/36df3073-cda4-425b-b552-02eb2b40a494-kube-api-access-fhlb7\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.215723 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.215508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-credential-socket\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.215723 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.215526 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36df3073-cda4-425b-b552-02eb2b40a494-istio-token\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.215723 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.215556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-istio-envoy\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.215723 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.215570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-istio-data\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.215723 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.215592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-workload-certs\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.215723 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.215655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36df3073-cda4-425b-b552-02eb2b40a494-istiod-ca-cert\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.216162 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.215872 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-workload-socket\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.216162 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.215905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-credential-socket\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.216162 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.215951 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-istio-data\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.216303 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.216194 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-workload-certs\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.216381 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.216358 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36df3073-cda4-425b-b552-02eb2b40a494-istiod-ca-cert\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.217914 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.217891 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-istio-envoy\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.218165 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.218147 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36df3073-cda4-425b-b552-02eb2b40a494-istio-podinfo\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.222846 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.222819 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36df3073-cda4-425b-b552-02eb2b40a494-istio-token\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.223165 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.223148 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhlb7\" (UniqueName: \"kubernetes.io/projected/36df3073-cda4-425b-b552-02eb2b40a494-kube-api-access-fhlb7\") pod \"e2e-inf-b-614db5-openshift-default-697699b899-b6xwv\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.381847 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.381735 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:56.516007 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.515875 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv"] Jul 22 20:40:56.518793 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:40:56.518763 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36df3073_cda4_425b_b552_02eb2b40a494.slice/crio-fd5ab479833bf8db1e7c29541d1fffc7f76f7bda6ffc62026c52f64de43d5a8e WatchSource:0}: Error finding container fd5ab479833bf8db1e7c29541d1fffc7f76f7bda6ffc62026c52f64de43d5a8e: Status 404 returned error can't find the container with id fd5ab479833bf8db1e7c29541d1fffc7f76f7bda6ffc62026c52f64de43d5a8e Jul 22 20:40:56.520747 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.520712 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:40:56.520849 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.520793 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:40:56.520849 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.520837 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:40:56.578099 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.578069 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:40:56.971040 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.970999 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:56.989905 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.989872 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" event={"ID":"36df3073-cda4-425b-b552-02eb2b40a494","Type":"ContainerStarted","Data":"a72b32e318c1d7322ac3647ce257583e99a209eed3d1b1a6bee29a1ae9816ef3"} Jul 22 20:40:56.990129 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:56.989923 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" event={"ID":"36df3073-cda4-425b-b552-02eb2b40a494","Type":"ContainerStarted","Data":"fd5ab479833bf8db1e7c29541d1fffc7f76f7bda6ffc62026c52f64de43d5a8e"} Jul 22 20:40:57.009247 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:57.009194 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" podStartSLOduration=1.009176109 podStartE2EDuration="1.009176109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:40:57.007791192 +0000 UTC m=+2357.520670899" watchObservedRunningTime="2026-07-22 20:40:57.009176109 +0000 UTC m=+2357.522055816" Jul 22 20:40:57.382507 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:57.382402 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:57.387524 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:57.387494 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:57.997352 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:57.997315 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:58.013650 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:58.013605 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:40:59.280450 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:59.280414 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:59.381025 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:59.380986 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:59.675150 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:59.675047 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:40:59.775669 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:40:59.775635 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:41:03.428362 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:03.428323 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:41:05.472733 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:05.472298 2575 patch_prober.go:28] interesting pod/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": read tcp 10.133.0.2:50536->10.133.0.34:15021: read: connection reset by peer" start-of-body= Jul 22 20:41:05.472733 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:05.472361 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": read tcp 10.133.0.2:50536->10.133.0.34:15021: read: connection reset by peer" Jul 22 20:41:06.038936 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:06.038901 2575 generic.go:356] "Generic (PLEG): container finished" podID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerID="ff5573b354cb84e04b78f127604c204b8d38491f11316b7f8645bfcdf947a4f9" exitCode=0 Jul 22 20:41:06.039146 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:06.038994 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" event={"ID":"cff36d44-7691-47e1-81fa-e826a9ce49dd","Type":"ContainerDied","Data":"ff5573b354cb84e04b78f127604c204b8d38491f11316b7f8645bfcdf947a4f9"} Jul 22 20:41:06.039486 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:06.039463 2575 scope.go:117] "RemoveContainer" containerID="ff5573b354cb84e04b78f127604c204b8d38491f11316b7f8645bfcdf947a4f9" Jul 22 20:41:06.040531 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:06.040493 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:41:06.040647 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:06.040568 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:41:06.040647 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:06.040608 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:41:07.045355 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:07.045317 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" event={"ID":"cff36d44-7691-47e1-81fa-e826a9ce49dd","Type":"ContainerStarted","Data":"a61c7f2c5e7b0d4951a26c944906ecb2de9887a643787e9cc83798da3e465f65"} Jul 22 20:41:07.916025 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:07.915668 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:41:07.916241 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:07.916135 2575 patch_prober.go:28] interesting pod/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 22 20:41:07.916241 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:07.916198 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 22 20:41:08.216953 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:08.216855 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:41:08.465778 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:08.465747 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:41:08.916140 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:08.916088 2575 patch_prober.go:28] interesting pod/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 22 20:41:08.916322 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:08.916159 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 22 20:41:09.916111 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:09.916001 2575 patch_prober.go:28] interesting pod/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 22 20:41:09.916111 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:09.916071 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 22 20:41:10.061118 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:10.061083 2575 generic.go:356] "Generic (PLEG): container finished" podID="36df3073-cda4-425b-b552-02eb2b40a494" containerID="a72b32e318c1d7322ac3647ce257583e99a209eed3d1b1a6bee29a1ae9816ef3" exitCode=0 Jul 22 20:41:10.061301 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:10.061158 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" event={"ID":"36df3073-cda4-425b-b552-02eb2b40a494","Type":"ContainerDied","Data":"a72b32e318c1d7322ac3647ce257583e99a209eed3d1b1a6bee29a1ae9816ef3"} Jul 22 20:41:10.061552 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:10.061532 2575 scope.go:117] "RemoveContainer" containerID="a72b32e318c1d7322ac3647ce257583e99a209eed3d1b1a6bee29a1ae9816ef3" Jul 22 20:41:10.062511 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:10.062481 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:41:10.062579 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:10.062555 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:41:10.062615 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:10.062602 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:41:10.915850 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:10.915810 2575 patch_prober.go:28] interesting pod/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 22 20:41:10.916063 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:10.915867 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 22 20:41:11.067951 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:11.067909 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" event={"ID":"36df3073-cda4-425b-b552-02eb2b40a494","Type":"ContainerStarted","Data":"36986d4992485160aa9ec59785b5bd38c95355f1e504c97665554c75032b452d"} Jul 22 20:41:11.382831 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:11.382788 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:41:11.383908 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:11.383876 2575 patch_prober.go:28] interesting pod/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 22 20:41:11.384034 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:11.383931 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" podUID="36df3073-cda4-425b-b552-02eb2b40a494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 22 20:41:11.920139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:11.920106 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:41:11.920520 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:11.920490 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:41:11.921209 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:11.921188 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:41:12.382835 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:12.382795 2575 patch_prober.go:28] interesting pod/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 22 20:41:12.383256 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:12.382870 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" podUID="36df3073-cda4-425b-b552-02eb2b40a494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 22 20:41:12.715055 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:12.715016 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:41:12.817613 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:12.817576 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:41:12.998084 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:12.997966 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:41:13.382303 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:13.382212 2575 patch_prober.go:28] interesting pod/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 22 20:41:13.382303 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:13.382272 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" podUID="36df3073-cda4-425b-b552-02eb2b40a494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 22 20:41:14.382420 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:14.382378 2575 patch_prober.go:28] interesting pod/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 22 20:41:14.382836 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:14.382437 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" podUID="36df3073-cda4-425b-b552-02eb2b40a494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 22 20:41:15.416616 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:15.416582 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:41:15.418156 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:15.418127 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:41:20.086990 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:20.086948 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:41:20.508495 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:20.508454 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:41:21.122703 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:21.122666 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:41:24.228606 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.228569 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg"] Jul 22 20:41:24.232024 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.232004 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.235274 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.235253 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-614db5\"/\"openshift-service-ca.crt\"" Jul 22 20:41:24.236094 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.236076 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-614db5\"/\"default-dockercfg-k2g72\"" Jul 22 20:41:24.236213 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.236190 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-614db5\"/\"test-model-614db5-kserve-self-signed-certs\"" Jul 22 20:41:24.236338 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.236217 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-614db5\"/\"kube-root-ca.crt\"" Jul 22 20:41:24.241780 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.241751 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg"] Jul 22 20:41:24.320722 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.320686 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:41:24.364343 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.364305 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-dshm\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.364509 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.364362 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-tmp-dir\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.364509 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.364457 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-home\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.364509 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.364486 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fb6859-9d39-4138-9dc1-827d7dccc5b9-tls-certs\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.364622 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.364565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-model-cache\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.364622 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.364598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh8th\" (UniqueName: \"kubernetes.io/projected/00fb6859-9d39-4138-9dc1-827d7dccc5b9-kube-api-access-sh8th\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.465594 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.465551 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-model-cache\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.465594 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.465598 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh8th\" (UniqueName: \"kubernetes.io/projected/00fb6859-9d39-4138-9dc1-827d7dccc5b9-kube-api-access-sh8th\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.465846 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.465628 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-dshm\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.465846 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.465660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-tmp-dir\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.465846 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.465697 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-home\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.465846 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.465719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fb6859-9d39-4138-9dc1-827d7dccc5b9-tls-certs\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.466076 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.466053 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-model-cache\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.466135 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.466096 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-tmp-dir\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.466212 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.466189 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-home\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.468019 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.467998 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-dshm\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.468296 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.468277 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fb6859-9d39-4138-9dc1-827d7dccc5b9-tls-certs\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.474878 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.474846 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh8th\" (UniqueName: \"kubernetes.io/projected/00fb6859-9d39-4138-9dc1-827d7dccc5b9-kube-api-access-sh8th\") pod \"test-model-614db5-kserve-598ddc8445-m9fxg\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.542450 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.542345 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:24.683323 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:24.683295 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg"] Jul 22 20:41:24.685860 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:41:24.685832 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00fb6859_9d39_4138_9dc1_827d7dccc5b9.slice/crio-c51a795d29b07e50e10211ce4580b1ae3f7ead281b7f68398f4e4e06316c73d6 WatchSource:0}: Error finding container c51a795d29b07e50e10211ce4580b1ae3f7ead281b7f68398f4e4e06316c73d6: Status 404 returned error can't find the container with id c51a795d29b07e50e10211ce4580b1ae3f7ead281b7f68398f4e4e06316c73d6 Jul 22 20:41:25.134399 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:25.134362 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" event={"ID":"00fb6859-9d39-4138-9dc1-827d7dccc5b9","Type":"ContainerStarted","Data":"5a4488cbe50dabf997014dd06032e71190089d6536ce414da400ee36d3315631"} Jul 22 20:41:25.134399 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:25.134405 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" event={"ID":"00fb6859-9d39-4138-9dc1-827d7dccc5b9","Type":"ContainerStarted","Data":"c51a795d29b07e50e10211ce4580b1ae3f7ead281b7f68398f4e4e06316c73d6"} Jul 22 20:41:25.150990 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:25.150919 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" podStartSLOduration=1.150901295 podStartE2EDuration="1.150901295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:41:25.15048433 +0000 UTC m=+2385.663364037" watchObservedRunningTime="2026-07-22 20:41:25.150901295 +0000 UTC m=+2385.663781000" Jul 22 20:41:30.567159 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:30.567122 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:41:34.543001 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:34.542954 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:34.543474 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:34.543013 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:34.555465 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:34.555437 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:35.191111 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:35.191082 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:41:40.135923 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:40.135891 2575 scope.go:117] "RemoveContainer" containerID="366930324caed8965f54d1182ba27e62d6bdf1ed0900901ab7fc1680e91ea1db" Jul 22 20:41:40.849018 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:40.848970 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:41:40.851014 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:41:40.850994 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:42:02.492245 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:02.492151 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:03.452851 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:03.452810 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg"] Jul 22 20:42:03.475383 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:03.475346 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:04.184546 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:04.184507 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:04.283405 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:04.283366 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:04.568118 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:04.568083 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:42:09.889972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:09.889925 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:14.086513 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:14.086473 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:16.772524 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:16.772479 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:42:18.473652 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.473613 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" podUID="00fb6859-9d39-4138-9dc1-827d7dccc5b9" containerName="main" containerID="cri-o://5a4488cbe50dabf997014dd06032e71190089d6536ce414da400ee36d3315631" gracePeriod=45 Jul 22 20:42:18.662957 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.662923 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:18.715270 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.715241 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:42:18.838052 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.837952 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh8th\" (UniqueName: \"kubernetes.io/projected/00fb6859-9d39-4138-9dc1-827d7dccc5b9-kube-api-access-sh8th\") pod \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " Jul 22 20:42:18.838052 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.838018 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fb6859-9d39-4138-9dc1-827d7dccc5b9-tls-certs\") pod \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " Jul 22 20:42:18.838052 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.838041 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-model-cache\") pod \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " Jul 22 20:42:18.838354 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.838085 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-tmp-dir\") pod \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " Jul 22 20:42:18.838354 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.838103 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-home\") pod \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " Jul 22 20:42:18.838354 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.838264 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-dshm\") pod \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\" (UID: \"00fb6859-9d39-4138-9dc1-827d7dccc5b9\") " Jul 22 20:42:18.838493 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.838372 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "00fb6859-9d39-4138-9dc1-827d7dccc5b9" (UID: "00fb6859-9d39-4138-9dc1-827d7dccc5b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:18.838493 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.838382 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00fb6859-9d39-4138-9dc1-827d7dccc5b9" (UID: "00fb6859-9d39-4138-9dc1-827d7dccc5b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:18.838493 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.838398 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-home" (OuterVolumeSpecName: "home") pod "00fb6859-9d39-4138-9dc1-827d7dccc5b9" (UID: "00fb6859-9d39-4138-9dc1-827d7dccc5b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:18.838626 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.838606 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-model-cache\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:18.838663 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.838636 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-tmp-dir\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:18.838663 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.838649 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-home\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:18.840338 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.840304 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-dshm" (OuterVolumeSpecName: "dshm") pod "00fb6859-9d39-4138-9dc1-827d7dccc5b9" (UID: "00fb6859-9d39-4138-9dc1-827d7dccc5b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:18.840445 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.840384 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fb6859-9d39-4138-9dc1-827d7dccc5b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00fb6859-9d39-4138-9dc1-827d7dccc5b9" (UID: "00fb6859-9d39-4138-9dc1-827d7dccc5b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 20:42:18.840500 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.840458 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fb6859-9d39-4138-9dc1-827d7dccc5b9-kube-api-access-sh8th" (OuterVolumeSpecName: "kube-api-access-sh8th") pod "00fb6859-9d39-4138-9dc1-827d7dccc5b9" (UID: "00fb6859-9d39-4138-9dc1-827d7dccc5b9"). InnerVolumeSpecName "kube-api-access-sh8th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:42:18.939424 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.939369 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fb6859-9d39-4138-9dc1-827d7dccc5b9-dshm\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:18.939424 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.939417 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh8th\" (UniqueName: \"kubernetes.io/projected/00fb6859-9d39-4138-9dc1-827d7dccc5b9-kube-api-access-sh8th\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:18.939424 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:18.939433 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fb6859-9d39-4138-9dc1-827d7dccc5b9-tls-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:19.048495 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:19.048453 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:19.369620 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:19.369584 2575 generic.go:356] "Generic (PLEG): container finished" podID="00fb6859-9d39-4138-9dc1-827d7dccc5b9" containerID="5a4488cbe50dabf997014dd06032e71190089d6536ce414da400ee36d3315631" exitCode=0 Jul 22 20:42:19.369796 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:19.369629 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" Jul 22 20:42:19.369796 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:19.369636 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" event={"ID":"00fb6859-9d39-4138-9dc1-827d7dccc5b9","Type":"ContainerDied","Data":"5a4488cbe50dabf997014dd06032e71190089d6536ce414da400ee36d3315631"} Jul 22 20:42:19.369796 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:19.369670 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg" event={"ID":"00fb6859-9d39-4138-9dc1-827d7dccc5b9","Type":"ContainerDied","Data":"c51a795d29b07e50e10211ce4580b1ae3f7ead281b7f68398f4e4e06316c73d6"} Jul 22 20:42:19.369796 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:19.369686 2575 scope.go:117] "RemoveContainer" containerID="5a4488cbe50dabf997014dd06032e71190089d6536ce414da400ee36d3315631" Jul 22 20:42:19.383596 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:19.383573 2575 scope.go:117] "RemoveContainer" containerID="5a4488cbe50dabf997014dd06032e71190089d6536ce414da400ee36d3315631" Jul 22 20:42:19.383901 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:42:19.383881 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a4488cbe50dabf997014dd06032e71190089d6536ce414da400ee36d3315631\": container with ID starting with 5a4488cbe50dabf997014dd06032e71190089d6536ce414da400ee36d3315631 not found: ID does not exist" containerID="5a4488cbe50dabf997014dd06032e71190089d6536ce414da400ee36d3315631" Jul 22 20:42:19.383953 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:19.383911 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a4488cbe50dabf997014dd06032e71190089d6536ce414da400ee36d3315631"} err="failed to get container status \"5a4488cbe50dabf997014dd06032e71190089d6536ce414da400ee36d3315631\": rpc error: code = NotFound desc = could not find container \"5a4488cbe50dabf997014dd06032e71190089d6536ce414da400ee36d3315631\": container with ID starting with 5a4488cbe50dabf997014dd06032e71190089d6536ce414da400ee36d3315631 not found: ID does not exist" Jul 22 20:42:19.392334 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:19.392303 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg"] Jul 22 20:42:19.394543 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:19.394515 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-614db5/test-model-614db5-kserve-598ddc8445-m9fxg"] Jul 22 20:42:20.007404 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:20.007372 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00fb6859-9d39-4138-9dc1-827d7dccc5b9" path="/var/lib/kubelet/pods/00fb6859-9d39-4138-9dc1-827d7dccc5b9/volumes" Jul 22 20:42:24.719857 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:24.719822 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:24.793342 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:24.793299 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:24.957769 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:24.957735 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k"] Jul 22 20:42:24.958045 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:24.958017 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerName="istio-proxy" containerID="cri-o://a61c7f2c5e7b0d4951a26c944906ecb2de9887a643787e9cc83798da3e465f65" gracePeriod=30 Jul 22 20:42:24.985324 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:24.985211 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:25.240226 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:25.240127 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:25.864928 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:25.864890 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:25.961813 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:25.961772 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:26.922568 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:26.922531 2575 patch_prober.go:28] interesting pod/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 20:42:26.922969 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:26.922607 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 20:42:30.219398 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.219368 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:42:30.342311 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.342197 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-workload-socket\") pod \"cff36d44-7691-47e1-81fa-e826a9ce49dd\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " Jul 22 20:42:30.342311 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.342265 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-data\") pod \"cff36d44-7691-47e1-81fa-e826a9ce49dd\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " Jul 22 20:42:30.342311 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.342291 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-workload-certs\") pod \"cff36d44-7691-47e1-81fa-e826a9ce49dd\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " Jul 22 20:42:30.342625 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.342324 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-envoy\") pod \"cff36d44-7691-47e1-81fa-e826a9ce49dd\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " Jul 22 20:42:30.342625 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.342369 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-token\") pod \"cff36d44-7691-47e1-81fa-e826a9ce49dd\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " Jul 22 20:42:30.342625 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.342405 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-podinfo\") pod \"cff36d44-7691-47e1-81fa-e826a9ce49dd\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " Jul 22 20:42:30.342625 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.342446 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff36d44-7691-47e1-81fa-e826a9ce49dd-istiod-ca-cert\") pod \"cff36d44-7691-47e1-81fa-e826a9ce49dd\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " Jul 22 20:42:30.342625 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.342477 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-credential-socket\") pod \"cff36d44-7691-47e1-81fa-e826a9ce49dd\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " Jul 22 20:42:30.342625 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.342549 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nltrd\" (UniqueName: \"kubernetes.io/projected/cff36d44-7691-47e1-81fa-e826a9ce49dd-kube-api-access-nltrd\") pod \"cff36d44-7691-47e1-81fa-e826a9ce49dd\" (UID: \"cff36d44-7691-47e1-81fa-e826a9ce49dd\") " Jul 22 20:42:30.342625 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.342569 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cff36d44-7691-47e1-81fa-e826a9ce49dd" (UID: "cff36d44-7691-47e1-81fa-e826a9ce49dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:30.342934 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.342664 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cff36d44-7691-47e1-81fa-e826a9ce49dd" (UID: "cff36d44-7691-47e1-81fa-e826a9ce49dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:30.342934 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.342832 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff36d44-7691-47e1-81fa-e826a9ce49dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cff36d44-7691-47e1-81fa-e826a9ce49dd" (UID: "cff36d44-7691-47e1-81fa-e826a9ce49dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:42:30.343073 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.343050 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-workload-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:30.343126 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.343077 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-workload-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:30.343126 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.343090 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff36d44-7691-47e1-81fa-e826a9ce49dd-istiod-ca-cert\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:30.343194 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.343159 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cff36d44-7691-47e1-81fa-e826a9ce49dd" (UID: "cff36d44-7691-47e1-81fa-e826a9ce49dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:30.343333 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.343311 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "cff36d44-7691-47e1-81fa-e826a9ce49dd" (UID: "cff36d44-7691-47e1-81fa-e826a9ce49dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:30.344809 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.344778 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff36d44-7691-47e1-81fa-e826a9ce49dd-kube-api-access-nltrd" (OuterVolumeSpecName: "kube-api-access-nltrd") pod "cff36d44-7691-47e1-81fa-e826a9ce49dd" (UID: "cff36d44-7691-47e1-81fa-e826a9ce49dd"). InnerVolumeSpecName "kube-api-access-nltrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:42:30.344928 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.344844 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cff36d44-7691-47e1-81fa-e826a9ce49dd" (UID: "cff36d44-7691-47e1-81fa-e826a9ce49dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 20:42:30.345111 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.345092 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "cff36d44-7691-47e1-81fa-e826a9ce49dd" (UID: "cff36d44-7691-47e1-81fa-e826a9ce49dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:42:30.345173 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.345104 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cff36d44-7691-47e1-81fa-e826a9ce49dd" (UID: "cff36d44-7691-47e1-81fa-e826a9ce49dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:30.418942 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.418866 2575 generic.go:356] "Generic (PLEG): container finished" podID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerID="a61c7f2c5e7b0d4951a26c944906ecb2de9887a643787e9cc83798da3e465f65" exitCode=0 Jul 22 20:42:30.419201 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.418945 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" event={"ID":"cff36d44-7691-47e1-81fa-e826a9ce49dd","Type":"ContainerDied","Data":"a61c7f2c5e7b0d4951a26c944906ecb2de9887a643787e9cc83798da3e465f65"} Jul 22 20:42:30.419201 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.419019 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" Jul 22 20:42:30.419201 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.419050 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k" event={"ID":"cff36d44-7691-47e1-81fa-e826a9ce49dd","Type":"ContainerDied","Data":"00a5b9d67d26f2fedc93a5a9616364a0737d2743e853aa164babe1f730af2c3d"} Jul 22 20:42:30.419201 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.419075 2575 scope.go:117] "RemoveContainer" containerID="a61c7f2c5e7b0d4951a26c944906ecb2de9887a643787e9cc83798da3e465f65" Jul 22 20:42:30.428506 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.428484 2575 scope.go:117] "RemoveContainer" containerID="ff5573b354cb84e04b78f127604c204b8d38491f11316b7f8645bfcdf947a4f9" Jul 22 20:42:30.437144 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.437119 2575 scope.go:117] "RemoveContainer" containerID="a61c7f2c5e7b0d4951a26c944906ecb2de9887a643787e9cc83798da3e465f65" Jul 22 20:42:30.437467 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:42:30.437443 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61c7f2c5e7b0d4951a26c944906ecb2de9887a643787e9cc83798da3e465f65\": container with ID starting with a61c7f2c5e7b0d4951a26c944906ecb2de9887a643787e9cc83798da3e465f65 not found: ID does not exist" containerID="a61c7f2c5e7b0d4951a26c944906ecb2de9887a643787e9cc83798da3e465f65" Jul 22 20:42:30.437548 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.437479 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61c7f2c5e7b0d4951a26c944906ecb2de9887a643787e9cc83798da3e465f65"} err="failed to get container status \"a61c7f2c5e7b0d4951a26c944906ecb2de9887a643787e9cc83798da3e465f65\": rpc error: code = NotFound desc = could not find container \"a61c7f2c5e7b0d4951a26c944906ecb2de9887a643787e9cc83798da3e465f65\": container with ID starting with a61c7f2c5e7b0d4951a26c944906ecb2de9887a643787e9cc83798da3e465f65 not found: ID does not exist" Jul 22 20:42:30.437548 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.437504 2575 scope.go:117] "RemoveContainer" containerID="ff5573b354cb84e04b78f127604c204b8d38491f11316b7f8645bfcdf947a4f9" Jul 22 20:42:30.437832 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:42:30.437810 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5573b354cb84e04b78f127604c204b8d38491f11316b7f8645bfcdf947a4f9\": container with ID starting with ff5573b354cb84e04b78f127604c204b8d38491f11316b7f8645bfcdf947a4f9 not found: ID does not exist" containerID="ff5573b354cb84e04b78f127604c204b8d38491f11316b7f8645bfcdf947a4f9" Jul 22 20:42:30.437865 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.437834 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5573b354cb84e04b78f127604c204b8d38491f11316b7f8645bfcdf947a4f9"} err="failed to get container status \"ff5573b354cb84e04b78f127604c204b8d38491f11316b7f8645bfcdf947a4f9\": rpc error: code = NotFound desc = could not find container \"ff5573b354cb84e04b78f127604c204b8d38491f11316b7f8645bfcdf947a4f9\": container with ID starting with ff5573b354cb84e04b78f127604c204b8d38491f11316b7f8645bfcdf947a4f9 not found: ID does not exist" Jul 22 20:42:30.441010 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.440969 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k"] Jul 22 20:42:30.444172 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.444152 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-podinfo\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:30.444272 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.444177 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-credential-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:30.444272 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.444191 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nltrd\" (UniqueName: \"kubernetes.io/projected/cff36d44-7691-47e1-81fa-e826a9ce49dd-kube-api-access-nltrd\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:30.444272 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.444200 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-data\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:30.444272 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.444209 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-envoy\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:30.444272 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.444211 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ba38bd-openshift-default-594d95d565-lsc4k"] Jul 22 20:42:30.444272 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:30.444216 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff36d44-7691-47e1-81fa-e826a9ce49dd-istio-token\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:32.006178 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:32.006139 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" path="/var/lib/kubelet/pods/cff36d44-7691-47e1-81fa-e826a9ce49dd/volumes" Jul 22 20:42:36.035313 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:36.035273 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:39.178034 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:39.178000 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:42:40.467872 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:40.467825 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:40.873043 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:40.872932 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:46.892938 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:46.892903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:47.004437 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:47.004403 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:47.197634 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:47.197597 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv"] Jul 22 20:42:47.197844 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:47.197822 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" podUID="36df3073-cda4-425b-b552-02eb2b40a494" containerName="istio-proxy" containerID="cri-o://36986d4992485160aa9ec59785b5bd38c95355f1e504c97665554c75032b452d" gracePeriod=30 Jul 22 20:42:47.241765 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:47.241700 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:47.496256 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:47.496156 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:47.881764 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:47.881663 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:48.129377 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.129342 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq"] Jul 22 20:42:48.129741 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.129621 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerName="istio-proxy" Jul 22 20:42:48.129741 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.129632 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerName="istio-proxy" Jul 22 20:42:48.129741 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.129640 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00fb6859-9d39-4138-9dc1-827d7dccc5b9" containerName="main" Jul 22 20:42:48.129741 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.129646 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="00fb6859-9d39-4138-9dc1-827d7dccc5b9" containerName="main" Jul 22 20:42:48.129741 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.129663 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerName="istio-proxy" Jul 22 20:42:48.129741 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.129669 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerName="istio-proxy" Jul 22 20:42:48.129741 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.129712 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerName="istio-proxy" Jul 22 20:42:48.129741 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.129719 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="00fb6859-9d39-4138-9dc1-827d7dccc5b9" containerName="main" Jul 22 20:42:48.132392 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.132339 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.134725 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.134702 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-83c37c-openshift-default-dockercfg-dgvzg\"" Jul 22 20:42:48.145139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.145112 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq"] Jul 22 20:42:48.165755 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.165672 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:48.184869 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.184837 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-workload-socket\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.184869 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.184868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-credential-socket\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.185085 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.184898 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-token\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.185085 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.184952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-workload-certs\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.185085 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.184997 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istiod-ca-cert\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.185085 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.185031 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-data\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.185085 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.185048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-podinfo\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.185085 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.185062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5xf6\" (UniqueName: \"kubernetes.io/projected/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-kube-api-access-t5xf6\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.185284 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.185133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-envoy\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.286385 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.286343 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istiod-ca-cert\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.286385 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.286386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-data\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.286628 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.286403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-podinfo\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.286628 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.286427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xf6\" (UniqueName: \"kubernetes.io/projected/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-kube-api-access-t5xf6\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.286628 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.286471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-envoy\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.286628 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.286513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-workload-socket\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.286628 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.286535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-credential-socket\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.286628 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.286579 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-token\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.286628 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.286622 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-workload-certs\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.287011 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.286815 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-data\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.287082 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.287067 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-workload-socket\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.287190 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.287166 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-credential-socket\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.287261 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.287214 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-workload-certs\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.287369 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.287350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istiod-ca-cert\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.288734 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.288707 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-envoy\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.289012 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.288971 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-podinfo\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.294809 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.294777 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-token\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.294915 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.294858 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xf6\" (UniqueName: \"kubernetes.io/projected/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-kube-api-access-t5xf6\") pod \"e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.441993 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.441953 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:48.546731 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.546682 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:48.579290 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.579260 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq"] Jul 22 20:42:48.585242 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:42:48.583787 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d74bb5_cf19_4ae4_9e4f_06ea0dd87d91.slice/crio-814fab6b50fde399b9debbef4dc4f9b26e80e821d4078fc84d0f10c99937db44 WatchSource:0}: Error finding container 814fab6b50fde399b9debbef4dc4f9b26e80e821d4078fc84d0f10c99937db44: Status 404 returned error can't find the container with id 814fab6b50fde399b9debbef4dc4f9b26e80e821d4078fc84d0f10c99937db44 Jul 22 20:42:48.586403 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.586357 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:42:48.586523 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.586452 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:42:48.586523 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:48.586513 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:42:49.511079 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:49.511044 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" event={"ID":"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91","Type":"ContainerStarted","Data":"be29edf233ee64f0d162a16497d2743aa91a20f6a6744cda72fad39a23ba40c3"} Jul 22 20:42:49.511446 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:49.511102 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" event={"ID":"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91","Type":"ContainerStarted","Data":"814fab6b50fde399b9debbef4dc4f9b26e80e821d4078fc84d0f10c99937db44"} Jul 22 20:42:49.532634 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:49.532577 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" podStartSLOduration=1.532562248 podStartE2EDuration="1.532562248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:42:49.530654464 +0000 UTC m=+2470.043534170" watchObservedRunningTime="2026-07-22 20:42:49.532562248 +0000 UTC m=+2470.045441954" Jul 22 20:42:50.442153 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:50.442112 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:50.447012 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:50.446966 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:50.515040 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:50.515009 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:50.516055 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:50.516026 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:42:51.453155 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:51.453110 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:51.543352 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:51.543309 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:51.613190 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:51.613150 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:52.057561 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.057515 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:52.159150 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.159119 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:52.259133 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.259085 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:52.445546 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.445508 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:42:52.521778 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.521749 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-credential-socket\") pod \"36df3073-cda4-425b-b552-02eb2b40a494\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " Jul 22 20:42:52.521778 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.521783 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36df3073-cda4-425b-b552-02eb2b40a494-istio-token\") pod \"36df3073-cda4-425b-b552-02eb2b40a494\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " Jul 22 20:42:52.522147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.521807 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-istio-envoy\") pod \"36df3073-cda4-425b-b552-02eb2b40a494\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " Jul 22 20:42:52.522147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.521848 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-workload-socket\") pod \"36df3073-cda4-425b-b552-02eb2b40a494\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " Jul 22 20:42:52.522147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.521875 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-workload-certs\") pod \"36df3073-cda4-425b-b552-02eb2b40a494\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " Jul 22 20:42:52.522147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.521900 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhlb7\" (UniqueName: \"kubernetes.io/projected/36df3073-cda4-425b-b552-02eb2b40a494-kube-api-access-fhlb7\") pod \"36df3073-cda4-425b-b552-02eb2b40a494\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " Jul 22 20:42:52.522147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.521925 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36df3073-cda4-425b-b552-02eb2b40a494-istio-podinfo\") pod \"36df3073-cda4-425b-b552-02eb2b40a494\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " Jul 22 20:42:52.522147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.521969 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36df3073-cda4-425b-b552-02eb2b40a494-istiod-ca-cert\") pod \"36df3073-cda4-425b-b552-02eb2b40a494\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " Jul 22 20:42:52.522147 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.522088 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-istio-data\") pod \"36df3073-cda4-425b-b552-02eb2b40a494\" (UID: \"36df3073-cda4-425b-b552-02eb2b40a494\") " Jul 22 20:42:52.522655 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.522158 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "36df3073-cda4-425b-b552-02eb2b40a494" (UID: "36df3073-cda4-425b-b552-02eb2b40a494"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:52.522655 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.522172 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "36df3073-cda4-425b-b552-02eb2b40a494" (UID: "36df3073-cda4-425b-b552-02eb2b40a494"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:52.522655 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.522394 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "36df3073-cda4-425b-b552-02eb2b40a494" (UID: "36df3073-cda4-425b-b552-02eb2b40a494"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:52.522655 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.522432 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-credential-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:52.522655 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.522453 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-workload-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:52.522915 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.522713 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36df3073-cda4-425b-b552-02eb2b40a494-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "36df3073-cda4-425b-b552-02eb2b40a494" (UID: "36df3073-cda4-425b-b552-02eb2b40a494"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:42:52.522961 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.522913 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-istio-data" (OuterVolumeSpecName: "istio-data") pod "36df3073-cda4-425b-b552-02eb2b40a494" (UID: "36df3073-cda4-425b-b552-02eb2b40a494"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:52.524322 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.524283 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36df3073-cda4-425b-b552-02eb2b40a494-istio-token" (OuterVolumeSpecName: "istio-token") pod "36df3073-cda4-425b-b552-02eb2b40a494" (UID: "36df3073-cda4-425b-b552-02eb2b40a494"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:42:52.524854 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.524830 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "36df3073-cda4-425b-b552-02eb2b40a494" (UID: "36df3073-cda4-425b-b552-02eb2b40a494"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:42:52.524962 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.524931 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36df3073-cda4-425b-b552-02eb2b40a494-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "36df3073-cda4-425b-b552-02eb2b40a494" (UID: "36df3073-cda4-425b-b552-02eb2b40a494"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 20:42:52.524962 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.524946 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36df3073-cda4-425b-b552-02eb2b40a494-kube-api-access-fhlb7" (OuterVolumeSpecName: "kube-api-access-fhlb7") pod "36df3073-cda4-425b-b552-02eb2b40a494" (UID: "36df3073-cda4-425b-b552-02eb2b40a494"). InnerVolumeSpecName "kube-api-access-fhlb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:42:52.527524 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.527496 2575 generic.go:356] "Generic (PLEG): container finished" podID="36df3073-cda4-425b-b552-02eb2b40a494" containerID="36986d4992485160aa9ec59785b5bd38c95355f1e504c97665554c75032b452d" exitCode=0 Jul 22 20:42:52.527608 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.527553 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" Jul 22 20:42:52.527608 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.527583 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" event={"ID":"36df3073-cda4-425b-b552-02eb2b40a494","Type":"ContainerDied","Data":"36986d4992485160aa9ec59785b5bd38c95355f1e504c97665554c75032b452d"} Jul 22 20:42:52.527690 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.527638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv" event={"ID":"36df3073-cda4-425b-b552-02eb2b40a494","Type":"ContainerDied","Data":"fd5ab479833bf8db1e7c29541d1fffc7f76f7bda6ffc62026c52f64de43d5a8e"} Jul 22 20:42:52.527725 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.527665 2575 scope.go:117] "RemoveContainer" containerID="36986d4992485160aa9ec59785b5bd38c95355f1e504c97665554c75032b452d" Jul 22 20:42:52.544339 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.544315 2575 scope.go:117] "RemoveContainer" containerID="a72b32e318c1d7322ac3647ce257583e99a209eed3d1b1a6bee29a1ae9816ef3" Jul 22 20:42:52.552601 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.552532 2575 scope.go:117] "RemoveContainer" containerID="36986d4992485160aa9ec59785b5bd38c95355f1e504c97665554c75032b452d" Jul 22 20:42:52.553451 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:42:52.553424 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36986d4992485160aa9ec59785b5bd38c95355f1e504c97665554c75032b452d\": container with ID starting with 36986d4992485160aa9ec59785b5bd38c95355f1e504c97665554c75032b452d not found: ID does not exist" containerID="36986d4992485160aa9ec59785b5bd38c95355f1e504c97665554c75032b452d" Jul 22 20:42:52.553549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.553457 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36986d4992485160aa9ec59785b5bd38c95355f1e504c97665554c75032b452d"} err="failed to get container status \"36986d4992485160aa9ec59785b5bd38c95355f1e504c97665554c75032b452d\": rpc error: code = NotFound desc = could not find container \"36986d4992485160aa9ec59785b5bd38c95355f1e504c97665554c75032b452d\": container with ID starting with 36986d4992485160aa9ec59785b5bd38c95355f1e504c97665554c75032b452d not found: ID does not exist" Jul 22 20:42:52.553549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.553476 2575 scope.go:117] "RemoveContainer" containerID="a72b32e318c1d7322ac3647ce257583e99a209eed3d1b1a6bee29a1ae9816ef3" Jul 22 20:42:52.553719 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:42:52.553673 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72b32e318c1d7322ac3647ce257583e99a209eed3d1b1a6bee29a1ae9816ef3\": container with ID starting with a72b32e318c1d7322ac3647ce257583e99a209eed3d1b1a6bee29a1ae9816ef3 not found: ID does not exist" containerID="a72b32e318c1d7322ac3647ce257583e99a209eed3d1b1a6bee29a1ae9816ef3" Jul 22 20:42:52.553719 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.553695 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72b32e318c1d7322ac3647ce257583e99a209eed3d1b1a6bee29a1ae9816ef3"} err="failed to get container status \"a72b32e318c1d7322ac3647ce257583e99a209eed3d1b1a6bee29a1ae9816ef3\": rpc error: code = NotFound desc = could not find container \"a72b32e318c1d7322ac3647ce257583e99a209eed3d1b1a6bee29a1ae9816ef3\": container with ID starting with a72b32e318c1d7322ac3647ce257583e99a209eed3d1b1a6bee29a1ae9816ef3 not found: ID does not exist" Jul 22 20:42:52.554634 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.554610 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv"] Jul 22 20:42:52.562906 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.562880 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-614db5-openshift-default-697699b899-b6xwv"] Jul 22 20:42:52.623715 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.623606 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-istio-data\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:52.623715 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.623642 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36df3073-cda4-425b-b552-02eb2b40a494-istio-token\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:52.623715 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.623655 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-istio-envoy\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:52.623715 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.623667 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36df3073-cda4-425b-b552-02eb2b40a494-workload-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:52.623715 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.623683 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhlb7\" (UniqueName: \"kubernetes.io/projected/36df3073-cda4-425b-b552-02eb2b40a494-kube-api-access-fhlb7\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:52.623715 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.623698 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36df3073-cda4-425b-b552-02eb2b40a494-istio-podinfo\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:52.623715 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:52.623710 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36df3073-cda4-425b-b552-02eb2b40a494-istiod-ca-cert\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:42:54.006652 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:54.006617 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36df3073-cda4-425b-b552-02eb2b40a494" path="/var/lib/kubelet/pods/36df3073-cda4-425b-b552-02eb2b40a494/volumes" Jul 22 20:42:55.961348 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:55.961310 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:56.444829 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.444798 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:56.733275 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.733189 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg"] Jul 22 20:42:56.733480 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.733467 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36df3073-cda4-425b-b552-02eb2b40a494" containerName="istio-proxy" Jul 22 20:42:56.733552 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.733481 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="36df3073-cda4-425b-b552-02eb2b40a494" containerName="istio-proxy" Jul 22 20:42:56.733552 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.733542 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="36df3073-cda4-425b-b552-02eb2b40a494" containerName="istio-proxy" Jul 22 20:42:56.733552 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.733551 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff36d44-7691-47e1-81fa-e826a9ce49dd" containerName="istio-proxy" Jul 22 20:42:56.733721 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.733558 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="36df3073-cda4-425b-b552-02eb2b40a494" containerName="istio-proxy" Jul 22 20:42:56.737831 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.737809 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.740105 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.740086 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3f3d31-openshift-default-dockercfg-zcwwr\"" Jul 22 20:42:56.749972 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.749949 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg"] Jul 22 20:42:56.757331 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.757293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-credential-socket\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.757431 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.757348 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33fc1f1-112c-4084-9174-f958250609e1-istiod-ca-cert\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.757431 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.757410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33fc1f1-112c-4084-9174-f958250609e1-istio-token\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.757503 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.757475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjpxb\" (UniqueName: \"kubernetes.io/projected/e33fc1f1-112c-4084-9174-f958250609e1-kube-api-access-qjpxb\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.757552 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.757511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-istio-data\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.757587 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.757548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-istio-envoy\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.757621 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.757597 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-workload-certs\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.757672 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.757654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33fc1f1-112c-4084-9174-f958250609e1-istio-podinfo\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.757721 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.757707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-workload-socket\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.769356 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.769315 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:56.859051 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859005 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-credential-socket\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.859051 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33fc1f1-112c-4084-9174-f958250609e1-istiod-ca-cert\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.859306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33fc1f1-112c-4084-9174-f958250609e1-istio-token\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.859306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjpxb\" (UniqueName: \"kubernetes.io/projected/e33fc1f1-112c-4084-9174-f958250609e1-kube-api-access-qjpxb\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.859306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-istio-data\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.859306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-istio-envoy\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.859306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-workload-certs\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.859306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33fc1f1-112c-4084-9174-f958250609e1-istio-podinfo\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.859306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859298 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-workload-socket\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.859647 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859493 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-credential-socket\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.859647 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859576 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-workload-certs\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.859754 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859711 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-istio-data\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.859833 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859811 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-workload-socket\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.859926 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.859909 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33fc1f1-112c-4084-9174-f958250609e1-istiod-ca-cert\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.861662 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.861641 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-istio-envoy\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.861884 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.861864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33fc1f1-112c-4084-9174-f958250609e1-istio-podinfo\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.868475 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.868450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33fc1f1-112c-4084-9174-f958250609e1-istio-token\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.868617 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.868598 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjpxb\" (UniqueName: \"kubernetes.io/projected/e33fc1f1-112c-4084-9174-f958250609e1-kube-api-access-qjpxb\") pod \"e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:56.883354 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:56.883330 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:57.047819 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:57.047729 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:57.179145 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:57.179121 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg"] Jul 22 20:42:57.182086 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:42:57.182056 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33fc1f1_112c_4084_9174_f958250609e1.slice/crio-1417a2b2b4a07ccb321f667a378c0858ec3c06477e63702b3e9b22ca824a806f WatchSource:0}: Error finding container 1417a2b2b4a07ccb321f667a378c0858ec3c06477e63702b3e9b22ca824a806f: Status 404 returned error can't find the container with id 1417a2b2b4a07ccb321f667a378c0858ec3c06477e63702b3e9b22ca824a806f Jul 22 20:42:57.182774 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:57.182751 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:42:57.184311 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:57.184279 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:42:57.184467 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:57.184339 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:42:57.184467 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:57.184366 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:42:57.554594 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:57.554552 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" event={"ID":"e33fc1f1-112c-4084-9174-f958250609e1","Type":"ContainerStarted","Data":"3399e24c1e68aa4fca1c18d99aa6d82fc56b0f86144378aa84596f2cf4e1a444"} Jul 22 20:42:57.554943 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:57.554620 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" event={"ID":"e33fc1f1-112c-4084-9174-f958250609e1","Type":"ContainerStarted","Data":"1417a2b2b4a07ccb321f667a378c0858ec3c06477e63702b3e9b22ca824a806f"} Jul 22 20:42:57.573276 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:57.573186 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" podStartSLOduration=1.573165111 podStartE2EDuration="1.573165111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:42:57.572120782 +0000 UTC m=+2478.085000491" watchObservedRunningTime="2026-07-22 20:42:57.573165111 +0000 UTC m=+2478.086044817" Jul 22 20:42:58.048217 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:58.048182 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:58.053096 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:58.053073 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:58.559323 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:58.559287 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:58.560362 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:58.560338 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:42:59.518161 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:59.518125 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:59.612805 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:59.612758 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:59.882579 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:59.882494 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:42:59.938496 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:42:59.938451 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:00.266076 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:00.266039 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:00.364489 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:00.364452 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:04.000756 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:04.000719 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:05.516610 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:05.516569 2575 patch_prober.go:28] interesting pod/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 22 20:43:05.517074 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:05.516643 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 22 20:43:05.591835 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:05.591796 2575 generic.go:356] "Generic (PLEG): container finished" podID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerID="be29edf233ee64f0d162a16497d2743aa91a20f6a6744cda72fad39a23ba40c3" exitCode=0 Jul 22 20:43:05.592184 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:05.591953 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" event={"ID":"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91","Type":"ContainerDied","Data":"be29edf233ee64f0d162a16497d2743aa91a20f6a6744cda72fad39a23ba40c3"} Jul 22 20:43:05.592778 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:05.592756 2575 scope.go:117] "RemoveContainer" containerID="be29edf233ee64f0d162a16497d2743aa91a20f6a6744cda72fad39a23ba40c3" Jul 22 20:43:05.595099 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:05.595045 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:43:05.595203 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:05.595170 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:43:05.595266 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:05.595233 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:43:06.602078 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:06.602035 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" event={"ID":"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91","Type":"ContainerStarted","Data":"2909ab316a0c013b075ba1366c166999da3c947f5b18d6a0024f1b085fb1b130"} Jul 22 20:43:07.442522 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:07.442480 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:43:07.442850 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:07.442808 2575 patch_prober.go:28] interesting pod/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 22 20:43:07.442944 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:07.442873 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 22 20:43:08.442550 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:08.442398 2575 patch_prober.go:28] interesting pod/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 22 20:43:08.442550 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:08.442468 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 22 20:43:09.442670 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:09.442628 2575 patch_prober.go:28] interesting pod/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 22 20:43:09.443073 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:09.442719 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 22 20:43:10.442941 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:10.442892 2575 patch_prober.go:28] interesting pod/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 22 20:43:10.443442 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:10.442998 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 22 20:43:10.475281 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:10.475232 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:43:11.448044 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:11.448000 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:43:11.448569 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:11.448287 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:43:11.448919 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:11.448899 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:43:12.180824 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.180783 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh"] Jul 22 20:43:12.181125 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.181111 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36df3073-cda4-425b-b552-02eb2b40a494" containerName="istio-proxy" Jul 22 20:43:12.181184 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.181130 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="36df3073-cda4-425b-b552-02eb2b40a494" containerName="istio-proxy" Jul 22 20:43:12.183150 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.183130 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh" Jul 22 20:43:12.185315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.185291 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjqt7\"" Jul 22 20:43:12.185432 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.185291 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 20:43:12.190549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.190523 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh"] Jul 22 20:43:12.300930 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.300841 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ecbce6-b758-4008-9f50-800e1a0acc55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh\" (UID: \"d2ecbce6-b758-4008-9f50-800e1a0acc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh" Jul 22 20:43:12.402408 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.402311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ecbce6-b758-4008-9f50-800e1a0acc55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh\" (UID: \"d2ecbce6-b758-4008-9f50-800e1a0acc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh" Jul 22 20:43:12.402960 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.402936 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ecbce6-b758-4008-9f50-800e1a0acc55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh\" (UID: \"d2ecbce6-b758-4008-9f50-800e1a0acc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh" Jul 22 20:43:12.495305 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.495263 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh" Jul 22 20:43:12.630826 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.630712 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh"] Jul 22 20:43:12.633749 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:43:12.633724 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ecbce6_b758_4008_9f50_800e1a0acc55.slice/crio-afdac07897fce9166952a48ac9fcd0c8d525230513a92dd93d85f09d52d08eb6 WatchSource:0}: Error finding container afdac07897fce9166952a48ac9fcd0c8d525230513a92dd93d85f09d52d08eb6: Status 404 returned error can't find the container with id afdac07897fce9166952a48ac9fcd0c8d525230513a92dd93d85f09d52d08eb6 Jul 22 20:43:12.639431 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.639407 2575 generic.go:356] "Generic (PLEG): container finished" podID="e33fc1f1-112c-4084-9174-f958250609e1" containerID="3399e24c1e68aa4fca1c18d99aa6d82fc56b0f86144378aa84596f2cf4e1a444" exitCode=0 Jul 22 20:43:12.639503 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.639475 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" event={"ID":"e33fc1f1-112c-4084-9174-f958250609e1","Type":"ContainerDied","Data":"3399e24c1e68aa4fca1c18d99aa6d82fc56b0f86144378aa84596f2cf4e1a444"} Jul 22 20:43:12.640211 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.640195 2575 scope.go:117] "RemoveContainer" containerID="3399e24c1e68aa4fca1c18d99aa6d82fc56b0f86144378aa84596f2cf4e1a444" Jul 22 20:43:12.643653 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.643620 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:43:12.643746 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.643709 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:43:12.643787 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.643770 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:43:12.746716 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.746638 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:12.848811 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:12.848762 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:13.559889 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:13.559852 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:43:13.646881 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:13.646779 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" event={"ID":"e33fc1f1-112c-4084-9174-f958250609e1","Type":"ContainerStarted","Data":"4e15cbfc2bc607fd5274bd5f4d02f7568b062528f521e5e86e2080bceb835775"} Jul 22 20:43:13.648354 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:13.648332 2575 generic.go:356] "Generic (PLEG): container finished" podID="d2ecbce6-b758-4008-9f50-800e1a0acc55" containerID="1fef0798aeaafcd456eacabb7ac269264f63fb2774827c38221dd5ac5b07b2c5" exitCode=0 Jul 22 20:43:13.648474 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:13.648368 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh" event={"ID":"d2ecbce6-b758-4008-9f50-800e1a0acc55","Type":"ContainerDied","Data":"1fef0798aeaafcd456eacabb7ac269264f63fb2774827c38221dd5ac5b07b2c5"} Jul 22 20:43:13.648474 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:13.648396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh" event={"ID":"d2ecbce6-b758-4008-9f50-800e1a0acc55","Type":"ContainerStarted","Data":"afdac07897fce9166952a48ac9fcd0c8d525230513a92dd93d85f09d52d08eb6"} Jul 22 20:43:14.048888 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:14.048846 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:43:14.050069 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:14.050042 2575 patch_prober.go:28] interesting pod/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 22 20:43:14.050191 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:14.050091 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" podUID="e33fc1f1-112c-4084-9174-f958250609e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 22 20:43:14.820340 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:14.820311 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh" Jul 22 20:43:14.928591 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:14.928503 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ecbce6-b758-4008-9f50-800e1a0acc55-maas-api-service-ca\") pod \"d2ecbce6-b758-4008-9f50-800e1a0acc55\" (UID: \"d2ecbce6-b758-4008-9f50-800e1a0acc55\") " Jul 22 20:43:14.928841 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:14.928813 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ecbce6-b758-4008-9f50-800e1a0acc55-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2ecbce6-b758-4008-9f50-800e1a0acc55" (UID: "d2ecbce6-b758-4008-9f50-800e1a0acc55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:43:15.029661 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:15.029613 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ecbce6-b758-4008-9f50-800e1a0acc55-maas-api-service-ca\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:43:15.048844 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:15.048804 2575 patch_prober.go:28] interesting pod/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 22 20:43:15.049062 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:15.048870 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" podUID="e33fc1f1-112c-4084-9174-f958250609e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 22 20:43:15.658290 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:15.658243 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh" event={"ID":"d2ecbce6-b758-4008-9f50-800e1a0acc55","Type":"ContainerDied","Data":"afdac07897fce9166952a48ac9fcd0c8d525230513a92dd93d85f09d52d08eb6"} Jul 22 20:43:15.658468 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:15.658347 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afdac07897fce9166952a48ac9fcd0c8d525230513a92dd93d85f09d52d08eb6" Jul 22 20:43:15.658468 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:15.658264 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh" Jul 22 20:43:16.048920 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:16.048870 2575 patch_prober.go:28] interesting pod/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 22 20:43:16.049358 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:16.048952 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" podUID="e33fc1f1-112c-4084-9174-f958250609e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 22 20:43:17.048811 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:17.048769 2575 patch_prober.go:28] interesting pod/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 22 20:43:17.049022 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:17.048843 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" podUID="e33fc1f1-112c-4084-9174-f958250609e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 22 20:43:18.052666 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:18.052633 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:43:18.053084 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:18.052907 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:43:18.053613 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:18.053592 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:43:21.021120 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:21.021088 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:21.122351 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:21.122318 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:22.823737 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:22.823682 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:22.921942 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:22.921894 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:28.066569 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:28.066530 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:30.561860 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:30.561826 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:43:32.250030 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:32.249919 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh"] Jul 22 20:43:32.251182 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:32.251158 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83c37c-d6wfh"] Jul 22 20:43:32.253935 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:32.253908 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:32.653576 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:32.653475 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:34.006356 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:34.006323 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2ecbce6-b758-4008-9f50-800e1a0acc55" path="/var/lib/kubelet/pods/d2ecbce6-b758-4008-9f50-800e1a0acc55/volumes" Jul 22 20:43:38.559125 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:38.559088 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:38.626576 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:38.626531 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:38.813759 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:38.813657 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq"] Jul 22 20:43:38.813961 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:38.813934 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerName="istio-proxy" containerID="cri-o://2909ab316a0c013b075ba1366c166999da3c947f5b18d6a0024f1b085fb1b130" gracePeriod=30 Jul 22 20:43:38.840219 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:38.840182 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:39.067516 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:39.067422 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:39.667716 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:39.667665 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:39.779661 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:39.779624 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:40.167698 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:40.167670 2575 scope.go:117] "RemoveContainer" containerID="742df2d86700963634969895afe7c453db634e6df3e43658d3c3604af60e0c27" Jul 22 20:43:41.450704 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:41.450665 2575 patch_prober.go:28] interesting pod/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 22 20:43:41.451100 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:41.450740 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 20:43:44.060967 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.060942 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:43:44.091205 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091122 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-workload-socket\") pod \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " Jul 22 20:43:44.091205 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091157 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-credential-socket\") pod \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " Jul 22 20:43:44.091205 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091186 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-podinfo\") pod \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " Jul 22 20:43:44.091479 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091217 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-token\") pod \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " Jul 22 20:43:44.091479 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091242 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-workload-certs\") pod \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " Jul 22 20:43:44.091479 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091268 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5xf6\" (UniqueName: \"kubernetes.io/projected/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-kube-api-access-t5xf6\") pod \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " Jul 22 20:43:44.091479 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091290 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-data\") pod \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " Jul 22 20:43:44.091479 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091328 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istiod-ca-cert\") pod \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " Jul 22 20:43:44.091479 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091407 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-envoy\") pod \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\" (UID: \"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91\") " Jul 22 20:43:44.091479 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091462 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" (UID: "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:43:44.091841 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091506 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" (UID: "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:43:44.091841 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091696 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-workload-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:43:44.091841 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091716 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-credential-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:43:44.091841 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091829 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" (UID: "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:43:44.092063 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.091890 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" (UID: "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:43:44.092301 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.092276 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" (UID: "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:43:44.093769 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.093735 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" (UID: "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:43:44.093971 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.093946 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-kube-api-access-t5xf6" (OuterVolumeSpecName: "kube-api-access-t5xf6") pod "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" (UID: "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91"). InnerVolumeSpecName "kube-api-access-t5xf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:43:44.094076 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.093969 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" (UID: "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 20:43:44.094136 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.094095 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" (UID: "f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:43:44.193143 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.193092 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-workload-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:43:44.193143 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.193135 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5xf6\" (UniqueName: \"kubernetes.io/projected/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-kube-api-access-t5xf6\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:43:44.193143 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.193146 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-data\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:43:44.193143 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.193154 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istiod-ca-cert\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:43:44.193143 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.193161 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-envoy\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:43:44.193455 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.193171 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-podinfo\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:43:44.193455 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.193178 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91-istio-token\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:43:44.796421 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.796385 2575 generic.go:356] "Generic (PLEG): container finished" podID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerID="2909ab316a0c013b075ba1366c166999da3c947f5b18d6a0024f1b085fb1b130" exitCode=0 Jul 22 20:43:44.796607 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.796432 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" Jul 22 20:43:44.796607 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.796465 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" event={"ID":"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91","Type":"ContainerDied","Data":"2909ab316a0c013b075ba1366c166999da3c947f5b18d6a0024f1b085fb1b130"} Jul 22 20:43:44.796607 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.796522 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq" event={"ID":"f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91","Type":"ContainerDied","Data":"814fab6b50fde399b9debbef4dc4f9b26e80e821d4078fc84d0f10c99937db44"} Jul 22 20:43:44.796607 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.796545 2575 scope.go:117] "RemoveContainer" containerID="2909ab316a0c013b075ba1366c166999da3c947f5b18d6a0024f1b085fb1b130" Jul 22 20:43:44.805811 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.805793 2575 scope.go:117] "RemoveContainer" containerID="be29edf233ee64f0d162a16497d2743aa91a20f6a6744cda72fad39a23ba40c3" Jul 22 20:43:44.813288 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.813265 2575 scope.go:117] "RemoveContainer" containerID="2909ab316a0c013b075ba1366c166999da3c947f5b18d6a0024f1b085fb1b130" Jul 22 20:43:44.813549 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:43:44.813530 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2909ab316a0c013b075ba1366c166999da3c947f5b18d6a0024f1b085fb1b130\": container with ID starting with 2909ab316a0c013b075ba1366c166999da3c947f5b18d6a0024f1b085fb1b130 not found: ID does not exist" containerID="2909ab316a0c013b075ba1366c166999da3c947f5b18d6a0024f1b085fb1b130" Jul 22 20:43:44.813613 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.813567 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2909ab316a0c013b075ba1366c166999da3c947f5b18d6a0024f1b085fb1b130"} err="failed to get container status \"2909ab316a0c013b075ba1366c166999da3c947f5b18d6a0024f1b085fb1b130\": rpc error: code = NotFound desc = could not find container \"2909ab316a0c013b075ba1366c166999da3c947f5b18d6a0024f1b085fb1b130\": container with ID starting with 2909ab316a0c013b075ba1366c166999da3c947f5b18d6a0024f1b085fb1b130 not found: ID does not exist" Jul 22 20:43:44.813613 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.813586 2575 scope.go:117] "RemoveContainer" containerID="be29edf233ee64f0d162a16497d2743aa91a20f6a6744cda72fad39a23ba40c3" Jul 22 20:43:44.813849 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:43:44.813829 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be29edf233ee64f0d162a16497d2743aa91a20f6a6744cda72fad39a23ba40c3\": container with ID starting with be29edf233ee64f0d162a16497d2743aa91a20f6a6744cda72fad39a23ba40c3 not found: ID does not exist" containerID="be29edf233ee64f0d162a16497d2743aa91a20f6a6744cda72fad39a23ba40c3" Jul 22 20:43:44.813948 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.813853 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be29edf233ee64f0d162a16497d2743aa91a20f6a6744cda72fad39a23ba40c3"} err="failed to get container status \"be29edf233ee64f0d162a16497d2743aa91a20f6a6744cda72fad39a23ba40c3\": rpc error: code = NotFound desc = could not find container \"be29edf233ee64f0d162a16497d2743aa91a20f6a6744cda72fad39a23ba40c3\": container with ID starting with be29edf233ee64f0d162a16497d2743aa91a20f6a6744cda72fad39a23ba40c3 not found: ID does not exist" Jul 22 20:43:44.820327 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.820304 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq"] Jul 22 20:43:44.824112 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:44.824089 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-83c37c-openshift-default-c847d789c-4tdmq"] Jul 22 20:43:46.006810 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:46.006779 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" path="/var/lib/kubelet/pods/f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91/volumes" Jul 22 20:43:54.843407 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:54.843362 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:59.176432 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:59.176394 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:43:59.263624 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:59.263593 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:43:59.671448 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:43:59.671413 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:05.515798 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:05.515760 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:05.756624 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:05.756585 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg"] Jul 22 20:44:05.756869 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:05.756840 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" podUID="e33fc1f1-112c-4084-9174-f958250609e1" containerName="istio-proxy" containerID="cri-o://4e15cbfc2bc607fd5274bd5f4d02f7568b062528f521e5e86e2080bceb835775" gracePeriod=30 Jul 22 20:44:06.707781 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.707739 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps"] Jul 22 20:44:06.708169 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.708061 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ecbce6-b758-4008-9f50-800e1a0acc55" containerName="revoke-keys" Jul 22 20:44:06.708169 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.708074 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ecbce6-b758-4008-9f50-800e1a0acc55" containerName="revoke-keys" Jul 22 20:44:06.708169 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.708089 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerName="istio-proxy" Jul 22 20:44:06.708169 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.708095 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerName="istio-proxy" Jul 22 20:44:06.708169 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.708118 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerName="istio-proxy" Jul 22 20:44:06.708169 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.708124 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerName="istio-proxy" Jul 22 20:44:06.708169 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.708166 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerName="istio-proxy" Jul 22 20:44:06.708390 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.708176 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ecbce6-b758-4008-9f50-800e1a0acc55" containerName="revoke-keys" Jul 22 20:44:06.711075 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.711052 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.713396 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.713377 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-b0cd1b-openshift-default-dockercfg-x6xm6\"" Jul 22 20:44:06.722631 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.722600 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps"] Jul 22 20:44:06.806106 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.806063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-workload-socket\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.806306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.806122 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c024094b-887e-4fc9-9a2c-abadc1db217c-istiod-ca-cert\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.806306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.806224 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-credential-socket\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.806306 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.806267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9blq\" (UniqueName: \"kubernetes.io/projected/c024094b-887e-4fc9-9a2c-abadc1db217c-kube-api-access-w9blq\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.806482 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.806317 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-podinfo\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.806482 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.806374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-token\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.806482 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.806404 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-envoy\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.806482 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.806428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-data\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.806482 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.806462 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-workload-certs\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.907690 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.907651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-credential-socket\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.907690 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.907690 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9blq\" (UniqueName: \"kubernetes.io/projected/c024094b-887e-4fc9-9a2c-abadc1db217c-kube-api-access-w9blq\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.907948 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.907713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-podinfo\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.907948 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.907742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-token\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.907948 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.907766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-envoy\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.907948 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.907788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-data\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.908189 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.907951 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-workload-certs\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.908189 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.908018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-workload-socket\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.908189 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.908086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c024094b-887e-4fc9-9a2c-abadc1db217c-istiod-ca-cert\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.908361 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.908289 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-data\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.908361 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.908329 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-credential-socket\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.908462 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.908362 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-workload-certs\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.908462 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.908446 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-workload-socket\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.908733 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.908713 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c024094b-887e-4fc9-9a2c-abadc1db217c-istiod-ca-cert\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.910139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.910118 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-envoy\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.910232 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.910222 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-podinfo\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.915640 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.915613 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9blq\" (UniqueName: \"kubernetes.io/projected/c024094b-887e-4fc9-9a2c-abadc1db217c-kube-api-access-w9blq\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.916059 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.916037 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-token\") pod \"e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:06.918061 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:06.918021 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:07.020634 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:07.020537 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:07.159913 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:07.159885 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps"] Jul 22 20:44:07.161611 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:44:07.161583 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc024094b_887e_4fc9_9a2c_abadc1db217c.slice/crio-4f75a3ab2159cccc12a9b9143b79326e7c23b69e5449257241788795b6fe86fb WatchSource:0}: Error finding container 4f75a3ab2159cccc12a9b9143b79326e7c23b69e5449257241788795b6fe86fb: Status 404 returned error can't find the container with id 4f75a3ab2159cccc12a9b9143b79326e7c23b69e5449257241788795b6fe86fb Jul 22 20:44:07.163641 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:07.163610 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:44:07.163724 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:07.163679 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:44:07.163724 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:07.163709 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:44:07.893993 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:07.893939 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" event={"ID":"c024094b-887e-4fc9-9a2c-abadc1db217c","Type":"ContainerStarted","Data":"520c2cd218e99c1a89eb2f093cf993d3bc03f5087ed50de939ba70277b09cda9"} Jul 22 20:44:07.894381 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:07.894025 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" event={"ID":"c024094b-887e-4fc9-9a2c-abadc1db217c","Type":"ContainerStarted","Data":"4f75a3ab2159cccc12a9b9143b79326e7c23b69e5449257241788795b6fe86fb"} Jul 22 20:44:07.915089 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:07.915037 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" podStartSLOduration=1.91502002 podStartE2EDuration="1.91502002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:44:07.913057818 +0000 UTC m=+2548.425937526" watchObservedRunningTime="2026-07-22 20:44:07.91502002 +0000 UTC m=+2548.427899725" Jul 22 20:44:08.021305 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:08.021266 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:08.033023 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:08.032992 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:08.899644 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:08.899615 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:08.900498 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:08.900480 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:09.873073 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:09.873039 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:09.967167 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:09.967127 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:10.088762 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:10.088717 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:10.503895 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:10.503855 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:10.602589 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:10.602554 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:10.703188 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:10.703150 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:10.910747 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:10.910713 2575 generic.go:356] "Generic (PLEG): container finished" podID="e33fc1f1-112c-4084-9174-f958250609e1" containerID="4e15cbfc2bc607fd5274bd5f4d02f7568b062528f521e5e86e2080bceb835775" exitCode=0 Jul 22 20:44:10.910940 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:10.910787 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" event={"ID":"e33fc1f1-112c-4084-9174-f958250609e1","Type":"ContainerDied","Data":"4e15cbfc2bc607fd5274bd5f4d02f7568b062528f521e5e86e2080bceb835775"} Jul 22 20:44:10.910940 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:10.910851 2575 scope.go:117] "RemoveContainer" containerID="3399e24c1e68aa4fca1c18d99aa6d82fc56b0f86144378aa84596f2cf4e1a444" Jul 22 20:44:11.012100 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.012060 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:44:11.145918 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.145821 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33fc1f1-112c-4084-9174-f958250609e1-istio-podinfo\") pod \"e33fc1f1-112c-4084-9174-f958250609e1\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " Jul 22 20:44:11.145918 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.145872 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjpxb\" (UniqueName: \"kubernetes.io/projected/e33fc1f1-112c-4084-9174-f958250609e1-kube-api-access-qjpxb\") pod \"e33fc1f1-112c-4084-9174-f958250609e1\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " Jul 22 20:44:11.145918 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.145901 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-istio-envoy\") pod \"e33fc1f1-112c-4084-9174-f958250609e1\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " Jul 22 20:44:11.146238 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.145928 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-istio-data\") pod \"e33fc1f1-112c-4084-9174-f958250609e1\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " Jul 22 20:44:11.146238 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.145970 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33fc1f1-112c-4084-9174-f958250609e1-istio-token\") pod \"e33fc1f1-112c-4084-9174-f958250609e1\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " Jul 22 20:44:11.146238 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.146041 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-credential-socket\") pod \"e33fc1f1-112c-4084-9174-f958250609e1\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " Jul 22 20:44:11.146238 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.146072 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-workload-socket\") pod \"e33fc1f1-112c-4084-9174-f958250609e1\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " Jul 22 20:44:11.146238 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.146113 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33fc1f1-112c-4084-9174-f958250609e1-istiod-ca-cert\") pod \"e33fc1f1-112c-4084-9174-f958250609e1\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " Jul 22 20:44:11.146238 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.146148 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-workload-certs\") pod \"e33fc1f1-112c-4084-9174-f958250609e1\" (UID: \"e33fc1f1-112c-4084-9174-f958250609e1\") " Jul 22 20:44:11.146525 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.146371 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e33fc1f1-112c-4084-9174-f958250609e1" (UID: "e33fc1f1-112c-4084-9174-f958250609e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:44:11.146666 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.146616 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e33fc1f1-112c-4084-9174-f958250609e1" (UID: "e33fc1f1-112c-4084-9174-f958250609e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:44:11.146785 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.146687 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33fc1f1-112c-4084-9174-f958250609e1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e33fc1f1-112c-4084-9174-f958250609e1" (UID: "e33fc1f1-112c-4084-9174-f958250609e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:44:11.146849 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.146781 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e33fc1f1-112c-4084-9174-f958250609e1" (UID: "e33fc1f1-112c-4084-9174-f958250609e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:44:11.147012 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.146970 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-istio-data" (OuterVolumeSpecName: "istio-data") pod "e33fc1f1-112c-4084-9174-f958250609e1" (UID: "e33fc1f1-112c-4084-9174-f958250609e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:44:11.148264 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.148238 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33fc1f1-112c-4084-9174-f958250609e1-kube-api-access-qjpxb" (OuterVolumeSpecName: "kube-api-access-qjpxb") pod "e33fc1f1-112c-4084-9174-f958250609e1" (UID: "e33fc1f1-112c-4084-9174-f958250609e1"). InnerVolumeSpecName "kube-api-access-qjpxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:44:11.148612 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.148597 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33fc1f1-112c-4084-9174-f958250609e1-istio-token" (OuterVolumeSpecName: "istio-token") pod "e33fc1f1-112c-4084-9174-f958250609e1" (UID: "e33fc1f1-112c-4084-9174-f958250609e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:44:11.148768 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.148751 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e33fc1f1-112c-4084-9174-f958250609e1" (UID: "e33fc1f1-112c-4084-9174-f958250609e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:44:11.148820 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.148760 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e33fc1f1-112c-4084-9174-f958250609e1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e33fc1f1-112c-4084-9174-f958250609e1" (UID: "e33fc1f1-112c-4084-9174-f958250609e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 20:44:11.247305 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.247257 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33fc1f1-112c-4084-9174-f958250609e1-istio-podinfo\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:44:11.247305 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.247301 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjpxb\" (UniqueName: \"kubernetes.io/projected/e33fc1f1-112c-4084-9174-f958250609e1-kube-api-access-qjpxb\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:44:11.247515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.247317 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-istio-envoy\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:44:11.247515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.247331 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-istio-data\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:44:11.247515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.247344 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33fc1f1-112c-4084-9174-f958250609e1-istio-token\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:44:11.247515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.247357 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-credential-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:44:11.247515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.247371 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-workload-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:44:11.247515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.247383 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33fc1f1-112c-4084-9174-f958250609e1-istiod-ca-cert\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:44:11.247515 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.247396 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33fc1f1-112c-4084-9174-f958250609e1-workload-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:44:11.916787 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.916759 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" Jul 22 20:44:11.916940 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.916758 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg" event={"ID":"e33fc1f1-112c-4084-9174-f958250609e1","Type":"ContainerDied","Data":"1417a2b2b4a07ccb321f667a378c0858ec3c06477e63702b3e9b22ca824a806f"} Jul 22 20:44:11.916940 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.916921 2575 scope.go:117] "RemoveContainer" containerID="4e15cbfc2bc607fd5274bd5f4d02f7568b062528f521e5e86e2080bceb835775" Jul 22 20:44:11.940014 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.939965 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg"] Jul 22 20:44:11.943267 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:11.943236 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3f3d31-openshift-default-5b6ccdbcbf-kmkwg"] Jul 22 20:44:12.006302 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:12.006270 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e33fc1f1-112c-4084-9174-f958250609e1" path="/var/lib/kubelet/pods/e33fc1f1-112c-4084-9174-f958250609e1/volumes" Jul 22 20:44:14.381901 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:14.381864 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:14.732427 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:14.732393 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:15.010960 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.010877 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd"] Jul 22 20:44:15.011305 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.011289 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33fc1f1-112c-4084-9174-f958250609e1" containerName="istio-proxy" Jul 22 20:44:15.011375 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.011308 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33fc1f1-112c-4084-9174-f958250609e1" containerName="istio-proxy" Jul 22 20:44:15.011375 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.011336 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33fc1f1-112c-4084-9174-f958250609e1" containerName="istio-proxy" Jul 22 20:44:15.011375 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.011344 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33fc1f1-112c-4084-9174-f958250609e1" containerName="istio-proxy" Jul 22 20:44:15.011519 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.011414 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d74bb5-cf19-4ae4-9e4f-06ea0dd87d91" containerName="istio-proxy" Jul 22 20:44:15.011519 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.011425 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33fc1f1-112c-4084-9174-f958250609e1" containerName="istio-proxy" Jul 22 20:44:15.011519 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.011435 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33fc1f1-112c-4084-9174-f958250609e1" containerName="istio-proxy" Jul 22 20:44:15.015829 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.015808 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.018144 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.018125 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-198019-openshift-default-dockercfg-t694c\"" Jul 22 20:44:15.024064 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.024039 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd"] Jul 22 20:44:15.041950 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.041921 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:15.184263 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.184232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-token\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.184438 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.184276 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppw8k\" (UniqueName: \"kubernetes.io/projected/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-kube-api-access-ppw8k\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.184438 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.184299 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-workload-certs\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.184438 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.184368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-credential-socket\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.184543 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.184430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-workload-socket\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.184543 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.184462 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-envoy\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.184543 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.184482 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istiod-ca-cert\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.184642 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.184546 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-podinfo\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.184642 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.184610 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-data\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.220689 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.220649 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:15.285767 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.285676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-data\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.285767 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.285735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-token\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.286026 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.285776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppw8k\" (UniqueName: \"kubernetes.io/projected/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-kube-api-access-ppw8k\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.286026 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.285805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-workload-certs\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.286026 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.285938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-credential-socket\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.286026 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.286010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-workload-socket\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.286239 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.286044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-envoy\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.286239 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.286075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istiod-ca-cert\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.286239 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.286113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-podinfo\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.286239 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.286192 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-data\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.286448 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.286239 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-workload-certs\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.286448 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.286335 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-credential-socket\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.286448 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.286405 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-workload-socket\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.286694 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.286676 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istiod-ca-cert\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.288217 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.288197 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-envoy\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.288526 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.288506 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-podinfo\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.293382 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.293358 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-token\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.293468 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.293434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppw8k\" (UniqueName: \"kubernetes.io/projected/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-kube-api-access-ppw8k\") pod \"e2e-shared-b-198019-openshift-default-85568fb999-qdmpd\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.327156 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.327118 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:15.459586 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.459556 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd"] Jul 22 20:44:15.461632 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:44:15.461599 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63ce6b8_1453_41f0_a6a5_7ed73732e3b1.slice/crio-5889c392d188cdb3b3d2ea6851acd5397d1985761d9d65b5997651c8e0a660f8 WatchSource:0}: Error finding container 5889c392d188cdb3b3d2ea6851acd5397d1985761d9d65b5997651c8e0a660f8: Status 404 returned error can't find the container with id 5889c392d188cdb3b3d2ea6851acd5397d1985761d9d65b5997651c8e0a660f8 Jul 22 20:44:15.463776 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.463745 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:44:15.463842 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.463812 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:44:15.463883 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.463840 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:44:15.935936 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.935897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" event={"ID":"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1","Type":"ContainerStarted","Data":"963c72e76d3e820489bb6a8fe9adafd450870233fecbb0ce82bf59aabb4e1a28"} Jul 22 20:44:15.936178 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.935964 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" event={"ID":"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1","Type":"ContainerStarted","Data":"5889c392d188cdb3b3d2ea6851acd5397d1985761d9d65b5997651c8e0a660f8"} Jul 22 20:44:15.959934 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:15.959881 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" podStartSLOduration=1.959864229 podStartE2EDuration="1.959864229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:44:15.958510625 +0000 UTC m=+2556.471390369" watchObservedRunningTime="2026-07-22 20:44:15.959864229 +0000 UTC m=+2556.472743935" Jul 22 20:44:16.327606 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:16.327564 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:16.332495 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:16.332468 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:16.940491 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:16.940461 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:16.941508 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:16.941489 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:17.573447 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:17.573410 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:44:17.895454 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:17.895355 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:17.994615 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:17.994572 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:18.169522 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:18.169419 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:18.359909 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:18.359874 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:18.613531 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:18.613489 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:18.778990 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:18.778935 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:22.405714 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:22.405666 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:24.941200 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:24.941153 2575 patch_prober.go:28] interesting pod/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 20:44:24.941620 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:24.941227 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 20:44:25.979838 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:25.979747 2575 generic.go:356] "Generic (PLEG): container finished" podID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerID="520c2cd218e99c1a89eb2f093cf993d3bc03f5087ed50de939ba70277b09cda9" exitCode=0 Jul 22 20:44:25.979838 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:25.979792 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" event={"ID":"c024094b-887e-4fc9-9a2c-abadc1db217c","Type":"ContainerDied","Data":"520c2cd218e99c1a89eb2f093cf993d3bc03f5087ed50de939ba70277b09cda9"} Jul 22 20:44:25.980315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:25.980192 2575 scope.go:117] "RemoveContainer" containerID="520c2cd218e99c1a89eb2f093cf993d3bc03f5087ed50de939ba70277b09cda9" Jul 22 20:44:25.981238 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:25.981209 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:44:25.981321 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:25.981281 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:44:25.981321 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:25.981312 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:44:26.986102 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:26.986058 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" event={"ID":"c024094b-887e-4fc9-9a2c-abadc1db217c","Type":"ContainerStarted","Data":"fcb095dc94b2289428f84ce4363685787b2c7e4bc393020431acb50c00ac72c5"} Jul 22 20:44:27.021567 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:27.021510 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:27.021567 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:27.021560 2575 patch_prober.go:28] interesting pod/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 22 20:44:27.021869 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:27.021614 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 22 20:44:28.021297 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:28.021256 2575 patch_prober.go:28] interesting pod/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 22 20:44:28.021957 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:28.021921 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 22 20:44:28.873456 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:28.873401 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:44:29.022051 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:29.022008 2575 patch_prober.go:28] interesting pod/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 22 20:44:29.023148 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:29.022658 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 22 20:44:30.021663 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:30.021626 2575 patch_prober.go:28] interesting pod/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 22 20:44:30.021875 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:30.021692 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 22 20:44:31.025186 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:31.025150 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:31.025681 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:31.025445 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:31.026305 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:31.026287 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:44:32.094532 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:32.094488 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:32.202343 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:32.202304 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:32.351864 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:32.350884 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:32.448256 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:32.448220 2575 patch_prober.go:28] interesting pod/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:47574->10.133.0.41:15021: read: connection reset by peer" start-of-body= Jul 22 20:44:32.448475 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:32.448305 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:47574->10.133.0.41:15021: read: connection reset by peer" Jul 22 20:44:33.020992 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:33.020954 2575 generic.go:356] "Generic (PLEG): container finished" podID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerID="963c72e76d3e820489bb6a8fe9adafd450870233fecbb0ce82bf59aabb4e1a28" exitCode=0 Jul 22 20:44:33.021158 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:33.021007 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" event={"ID":"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1","Type":"ContainerDied","Data":"963c72e76d3e820489bb6a8fe9adafd450870233fecbb0ce82bf59aabb4e1a28"} Jul 22 20:44:33.021713 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:33.021686 2575 scope.go:117] "RemoveContainer" containerID="963c72e76d3e820489bb6a8fe9adafd450870233fecbb0ce82bf59aabb4e1a28" Jul 22 20:44:33.022763 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:33.022734 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:44:33.022872 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:33.022820 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:44:33.022937 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:33.022871 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 20:44:34.028649 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:34.028616 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" event={"ID":"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1","Type":"ContainerStarted","Data":"b69d80e97f1e0717d9efe884f7298b4334ee49944175b839bd0c2dd82e774fd8"} Jul 22 20:44:34.327804 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:34.327698 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:34.328051 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:34.328025 2575 patch_prober.go:28] interesting pod/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 20:44:34.328101 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:34.328080 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 20:44:35.328015 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:35.327963 2575 patch_prober.go:28] interesting pod/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 20:44:35.328397 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:35.328042 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 20:44:36.327665 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:36.327615 2575 patch_prober.go:28] interesting pod/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 20:44:36.327849 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:36.327683 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 20:44:36.565504 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:36.565468 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:44:37.327893 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:37.327844 2575 patch_prober.go:28] interesting pod/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 20:44:37.328122 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:37.327932 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 20:44:38.331006 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:38.330959 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:38.331468 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:38.331374 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:38.332204 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:38.332187 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:44:39.565478 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:39.565446 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:39.668291 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:39.668259 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:48.817137 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.817103 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm"] Jul 22 20:44:48.820273 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.820247 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:48.823549 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.823524 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-198019\"/\"auth-test-model-198019-kserve-self-signed-certs\"" Jul 22 20:44:48.823675 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.823529 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-198019\"/\"kube-root-ca.crt\"" Jul 22 20:44:48.824215 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.823865 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-198019\"/\"openshift-service-ca.crt\"" Jul 22 20:44:48.829434 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.825373 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-198019\"/\"default-dockercfg-fcqr6\"" Jul 22 20:44:48.831184 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.831158 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm"] Jul 22 20:44:48.849990 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.849947 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:44:48.898081 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.898044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-home\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:48.898253 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.898101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-tmp-dir\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:48.898253 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.898182 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nmwp\" (UniqueName: \"kubernetes.io/projected/09472f05-2338-4cfe-b846-b826f120a720-kube-api-access-4nmwp\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:48.898253 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.898217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09472f05-2338-4cfe-b846-b826f120a720-tls-certs\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:48.898380 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.898254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-model-cache\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:48.898380 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.898286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-dshm\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:48.999752 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.999708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-tmp-dir\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:48.999938 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.999796 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nmwp\" (UniqueName: \"kubernetes.io/projected/09472f05-2338-4cfe-b846-b826f120a720-kube-api-access-4nmwp\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:48.999938 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:48.999842 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09472f05-2338-4cfe-b846-b826f120a720-tls-certs\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:49.000116 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:49.000094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-model-cache\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:49.000226 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:49.000209 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-dshm\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:49.000401 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:49.000384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-home\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:49.000541 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:49.000432 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-model-cache\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:49.000645 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:49.000243 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-tmp-dir\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:49.000754 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:49.000650 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-home\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:49.003218 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:49.003193 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-dshm\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:49.003796 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:49.003776 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09472f05-2338-4cfe-b846-b826f120a720-tls-certs\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:49.010173 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:49.010145 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nmwp\" (UniqueName: \"kubernetes.io/projected/09472f05-2338-4cfe-b846-b826f120a720-kube-api-access-4nmwp\") pod \"auth-test-model-198019-kserve-5f879bf8b7-vsrdm\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:49.133113 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:49.133019 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:49.256274 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:49.256244 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm"] Jul 22 20:44:49.258494 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:44:49.258466 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09472f05_2338_4cfe_b846_b826f120a720.slice/crio-05c15b8e5622e3a6ae9a796172033625865e704088a097fb68c4f38aa6479843 WatchSource:0}: Error finding container 05c15b8e5622e3a6ae9a796172033625865e704088a097fb68c4f38aa6479843: Status 404 returned error can't find the container with id 05c15b8e5622e3a6ae9a796172033625865e704088a097fb68c4f38aa6479843 Jul 22 20:44:50.094324 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:50.094289 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" event={"ID":"09472f05-2338-4cfe-b846-b826f120a720","Type":"ContainerStarted","Data":"869bd69c7c2cead339ef3ececcaf990346036eafa24a4adac0915556340341b7"} Jul 22 20:44:50.094324 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:50.094333 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" event={"ID":"09472f05-2338-4cfe-b846-b826f120a720","Type":"ContainerStarted","Data":"05c15b8e5622e3a6ae9a796172033625865e704088a097fb68c4f38aa6479843"} Jul 22 20:44:50.110932 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:50.110887 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" podStartSLOduration=2.11087017 podStartE2EDuration="2.11087017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:44:50.109779359 +0000 UTC m=+2590.622659066" watchObservedRunningTime="2026-07-22 20:44:50.11087017 +0000 UTC m=+2590.623749875" Jul 22 20:44:52.074658 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:52.074622 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:44:59.133942 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:59.133895 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:59.134433 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:59.133999 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:44:59.147049 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:44:59.147020 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:45:00.138146 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:00.138111 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-198019-29745885-h7cc5"] Jul 22 20:45:00.141518 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:00.141496 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-198019-29745885-h7cc5" Jul 22 20:45:00.143462 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:00.143442 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjqt7\"" Jul 22 20:45:00.151185 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:00.151161 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-198019-29745885-h7cc5"] Jul 22 20:45:00.154248 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:00.154225 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-198019-29745885-h7cc5" Jul 22 20:45:00.156288 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:00.156266 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:45:00.292248 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:00.292173 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-198019-29745885-h7cc5"] Jul 22 20:45:00.294518 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:45:00.294491 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ab4cdd_f494_4a7e_949e_0d848fb3349a.slice/crio-5a3435bd2a9876decca577ae1265102f5a203503adc177bcdf333a0adc50bad8 WatchSource:0}: Error finding container 5a3435bd2a9876decca577ae1265102f5a203503adc177bcdf333a0adc50bad8: Status 404 returned error can't find the container with id 5a3435bd2a9876decca577ae1265102f5a203503adc177bcdf333a0adc50bad8 Jul 22 20:45:01.147494 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:01.147461 2575 generic.go:356] "Generic (PLEG): container finished" podID="e4ab4cdd-f494-4a7e-949e-0d848fb3349a" containerID="6b02486329d32f5315892d3d73e7dc9f23d5aafdab5b553f70635b36b9d98cf0" exitCode=0 Jul 22 20:45:01.147853 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:01.147535 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-198019-29745885-h7cc5" event={"ID":"e4ab4cdd-f494-4a7e-949e-0d848fb3349a","Type":"ContainerDied","Data":"6b02486329d32f5315892d3d73e7dc9f23d5aafdab5b553f70635b36b9d98cf0"} Jul 22 20:45:01.147853 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:01.147588 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-198019-29745885-h7cc5" event={"ID":"e4ab4cdd-f494-4a7e-949e-0d848fb3349a","Type":"ContainerStarted","Data":"5a3435bd2a9876decca577ae1265102f5a203503adc177bcdf333a0adc50bad8"} Jul 22 20:45:02.276917 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:02.276892 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-198019-29745885-h7cc5" Jul 22 20:45:03.158096 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:03.158067 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-198019-29745885-h7cc5" Jul 22 20:45:03.158096 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:03.158083 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-198019-29745885-h7cc5" event={"ID":"e4ab4cdd-f494-4a7e-949e-0d848fb3349a","Type":"ContainerDied","Data":"5a3435bd2a9876decca577ae1265102f5a203503adc177bcdf333a0adc50bad8"} Jul 22 20:45:03.158309 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:03.158130 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a3435bd2a9876decca577ae1265102f5a203503adc177bcdf333a0adc50bad8" Jul 22 20:45:07.269801 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:07.269755 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:45:07.371108 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:07.371077 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:45:08.572941 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:08.572906 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:45:27.370686 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:27.370645 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:45:36.097036 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:36.097001 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:45:36.197407 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:36.197371 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:45:41.671712 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:45:41.671675 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:46:00.872187 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:46:00.872152 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:46:11.978056 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:46:11.978017 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:46:16.676412 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:46:16.676375 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:46:34.320186 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:46:34.320109 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:46:34.975082 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:46:34.975046 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:46:40.872477 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:46:40.872446 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:46:40.874783 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:46:40.874763 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsbtd_ce1c3d0f-7d3c-470e-8ef1-d20299d5d75b/ovn-acl-logging/0.log" Jul 22 20:46:51.771963 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:46:51.771921 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:46:56.485471 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:46:56.485436 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:46:57.410476 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:46:57.410439 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm"] Jul 22 20:46:57.430762 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:46:57.430725 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:47:07.268404 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:07.268367 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:47:08.945566 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:08.945524 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:47:12.430471 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.430430 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" podUID="09472f05-2338-4cfe-b846-b826f120a720" containerName="main" containerID="cri-o://869bd69c7c2cead339ef3ececcaf990346036eafa24a4adac0915556340341b7" gracePeriod=45 Jul 22 20:47:12.694031 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.694003 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:47:12.717032 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.717001 2575 generic.go:356] "Generic (PLEG): container finished" podID="09472f05-2338-4cfe-b846-b826f120a720" containerID="869bd69c7c2cead339ef3ececcaf990346036eafa24a4adac0915556340341b7" exitCode=0 Jul 22 20:47:12.717198 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.717041 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" Jul 22 20:47:12.717198 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.717081 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" event={"ID":"09472f05-2338-4cfe-b846-b826f120a720","Type":"ContainerDied","Data":"869bd69c7c2cead339ef3ececcaf990346036eafa24a4adac0915556340341b7"} Jul 22 20:47:12.717198 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.717119 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm" event={"ID":"09472f05-2338-4cfe-b846-b826f120a720","Type":"ContainerDied","Data":"05c15b8e5622e3a6ae9a796172033625865e704088a097fb68c4f38aa6479843"} Jul 22 20:47:12.717198 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.717137 2575 scope.go:117] "RemoveContainer" containerID="869bd69c7c2cead339ef3ececcaf990346036eafa24a4adac0915556340341b7" Jul 22 20:47:12.725209 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.725194 2575 scope.go:117] "RemoveContainer" containerID="869bd69c7c2cead339ef3ececcaf990346036eafa24a4adac0915556340341b7" Jul 22 20:47:12.725464 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:47:12.725447 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869bd69c7c2cead339ef3ececcaf990346036eafa24a4adac0915556340341b7\": container with ID starting with 869bd69c7c2cead339ef3ececcaf990346036eafa24a4adac0915556340341b7 not found: ID does not exist" containerID="869bd69c7c2cead339ef3ececcaf990346036eafa24a4adac0915556340341b7" Jul 22 20:47:12.725519 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.725474 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869bd69c7c2cead339ef3ececcaf990346036eafa24a4adac0915556340341b7"} err="failed to get container status \"869bd69c7c2cead339ef3ececcaf990346036eafa24a4adac0915556340341b7\": rpc error: code = NotFound desc = could not find container \"869bd69c7c2cead339ef3ececcaf990346036eafa24a4adac0915556340341b7\": container with ID starting with 869bd69c7c2cead339ef3ececcaf990346036eafa24a4adac0915556340341b7 not found: ID does not exist" Jul 22 20:47:12.748776 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.748753 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-tmp-dir\") pod \"09472f05-2338-4cfe-b846-b826f120a720\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " Jul 22 20:47:12.748905 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.748845 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nmwp\" (UniqueName: \"kubernetes.io/projected/09472f05-2338-4cfe-b846-b826f120a720-kube-api-access-4nmwp\") pod \"09472f05-2338-4cfe-b846-b826f120a720\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " Jul 22 20:47:12.748905 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.748889 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-dshm\") pod \"09472f05-2338-4cfe-b846-b826f120a720\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " Jul 22 20:47:12.749103 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.748917 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-home\") pod \"09472f05-2338-4cfe-b846-b826f120a720\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " Jul 22 20:47:12.749103 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.748968 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-model-cache\") pod \"09472f05-2338-4cfe-b846-b826f120a720\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " Jul 22 20:47:12.749103 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.748995 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09472f05-2338-4cfe-b846-b826f120a720" (UID: "09472f05-2338-4cfe-b846-b826f120a720"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:47:12.749103 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.749027 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09472f05-2338-4cfe-b846-b826f120a720-tls-certs\") pod \"09472f05-2338-4cfe-b846-b826f120a720\" (UID: \"09472f05-2338-4cfe-b846-b826f120a720\") " Jul 22 20:47:12.749313 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.749158 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-home" (OuterVolumeSpecName: "home") pod "09472f05-2338-4cfe-b846-b826f120a720" (UID: "09472f05-2338-4cfe-b846-b826f120a720"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:47:12.749313 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.749230 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-model-cache" (OuterVolumeSpecName: "model-cache") pod "09472f05-2338-4cfe-b846-b826f120a720" (UID: "09472f05-2338-4cfe-b846-b826f120a720"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:47:12.749406 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.749356 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-tmp-dir\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:47:12.749406 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.749374 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-home\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:47:12.749406 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.749384 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-model-cache\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:47:12.751233 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.751214 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09472f05-2338-4cfe-b846-b826f120a720-kube-api-access-4nmwp" (OuterVolumeSpecName: "kube-api-access-4nmwp") pod "09472f05-2338-4cfe-b846-b826f120a720" (UID: "09472f05-2338-4cfe-b846-b826f120a720"). InnerVolumeSpecName "kube-api-access-4nmwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:47:12.751420 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.751405 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09472f05-2338-4cfe-b846-b826f120a720-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09472f05-2338-4cfe-b846-b826f120a720" (UID: "09472f05-2338-4cfe-b846-b826f120a720"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 20:47:12.751473 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.751422 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-dshm" (OuterVolumeSpecName: "dshm") pod "09472f05-2338-4cfe-b846-b826f120a720" (UID: "09472f05-2338-4cfe-b846-b826f120a720"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:47:12.850771 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.850732 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nmwp\" (UniqueName: \"kubernetes.io/projected/09472f05-2338-4cfe-b846-b826f120a720-kube-api-access-4nmwp\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:47:12.850771 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.850763 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09472f05-2338-4cfe-b846-b826f120a720-dshm\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:47:12.850771 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:12.850787 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09472f05-2338-4cfe-b846-b826f120a720-tls-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:47:13.041452 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:13.041416 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm"] Jul 22 20:47:13.042845 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:13.042822 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-198019/auth-test-model-198019-kserve-5f879bf8b7-vsrdm"] Jul 22 20:47:14.006656 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:14.006620 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09472f05-2338-4cfe-b846-b826f120a720" path="/var/lib/kubelet/pods/09472f05-2338-4cfe-b846-b826f120a720/volumes" Jul 22 20:47:19.171207 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:19.171164 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:47:35.424218 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.424183 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8"] Jul 22 20:47:35.424627 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.424550 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09472f05-2338-4cfe-b846-b826f120a720" containerName="main" Jul 22 20:47:35.424627 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.424567 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="09472f05-2338-4cfe-b846-b826f120a720" containerName="main" Jul 22 20:47:35.424627 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.424580 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4ab4cdd-f494-4a7e-949e-0d848fb3349a" containerName="cleanup" Jul 22 20:47:35.424627 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.424585 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4ab4cdd-f494-4a7e-949e-0d848fb3349a" containerName="cleanup" Jul 22 20:47:35.424750 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.424655 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="09472f05-2338-4cfe-b846-b826f120a720" containerName="main" Jul 22 20:47:35.424750 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.424664 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4ab4cdd-f494-4a7e-949e-0d848fb3349a" containerName="cleanup" Jul 22 20:47:35.427703 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.427686 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.430319 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.430292 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-198019\"/\"openshift-service-ca.crt\"" Jul 22 20:47:35.431188 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.431170 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-198019\"/\"default-dockercfg-fcqr6\"" Jul 22 20:47:35.431293 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.431229 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-198019\"/\"sub-test-model-198019-kserve-self-signed-certs\"" Jul 22 20:47:35.431293 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.431261 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-198019\"/\"kube-root-ca.crt\"" Jul 22 20:47:35.435744 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.435700 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8"] Jul 22 20:47:35.444428 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.444403 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-tmp-dir\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.444557 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.444459 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8780312-da91-4ae5-9f16-fec199c058d7-tls-certs\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.444557 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.444488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkfhc\" (UniqueName: \"kubernetes.io/projected/a8780312-da91-4ae5-9f16-fec199c058d7-kube-api-access-kkfhc\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.444691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.444584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-home\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.444691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.444635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-model-cache\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.444691 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.444665 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-dshm\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.455239 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.455206 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:47:35.545131 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.545102 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-home\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.545318 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.545143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-model-cache\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.545318 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.545162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-dshm\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.545318 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.545205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-tmp-dir\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.545318 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.545231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8780312-da91-4ae5-9f16-fec199c058d7-tls-certs\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.545318 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.545247 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkfhc\" (UniqueName: \"kubernetes.io/projected/a8780312-da91-4ae5-9f16-fec199c058d7-kube-api-access-kkfhc\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.545625 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.545604 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-tmp-dir\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.545673 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.545601 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-model-cache\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.545713 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.545678 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-home\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.547599 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.547576 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-dshm\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.547731 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.547717 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8780312-da91-4ae5-9f16-fec199c058d7-tls-certs\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.552239 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.552223 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkfhc\" (UniqueName: \"kubernetes.io/projected/a8780312-da91-4ae5-9f16-fec199c058d7-kube-api-access-kkfhc\") pod \"sub-test-model-198019-kserve-94b6869c5-trpm8\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.737304 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.737273 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:35.866831 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:35.866792 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8"] Jul 22 20:47:35.868428 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:47:35.868401 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8780312_da91_4ae5_9f16_fec199c058d7.slice/crio-78012ccdfe84ced258fa32485dbb9ae041f3bb45e958b49336b54b9982b8c024 WatchSource:0}: Error finding container 78012ccdfe84ced258fa32485dbb9ae041f3bb45e958b49336b54b9982b8c024: Status 404 returned error can't find the container with id 78012ccdfe84ced258fa32485dbb9ae041f3bb45e958b49336b54b9982b8c024 Jul 22 20:47:36.819224 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:36.819186 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" event={"ID":"a8780312-da91-4ae5-9f16-fec199c058d7","Type":"ContainerStarted","Data":"4eed3bafd6e4db98b89cd8bde933ddb9919ba112d2ba00d20f27e2d99e082121"} Jul 22 20:47:36.819636 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:36.819230 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" event={"ID":"a8780312-da91-4ae5-9f16-fec199c058d7","Type":"ContainerStarted","Data":"78012ccdfe84ced258fa32485dbb9ae041f3bb45e958b49336b54b9982b8c024"} Jul 22 20:47:36.835328 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:36.835275 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" podStartSLOduration=1.835259353 podStartE2EDuration="1.835259353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:47:36.834290758 +0000 UTC m=+2757.347170464" watchObservedRunningTime="2026-07-22 20:47:36.835259353 +0000 UTC m=+2757.348139059" Jul 22 20:47:39.565851 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:39.565818 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:47:45.738117 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:45.738076 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:45.738613 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:45.738144 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:45.750637 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:45.750610 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:45.867060 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:45.867031 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:47:56.977398 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:47:56.977338 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:48:15.372416 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:15.372371 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:48:22.078692 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:22.078648 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:48:22.630897 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:22.630859 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8"] Jul 22 20:48:22.655131 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:22.655095 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:48:27.023961 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:27.023928 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:48:30.526759 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.526722 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2"] Jul 22 20:48:30.529969 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.529952 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.533261 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.533240 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b0cd1b\"/\"kube-root-ca.crt\"" Jul 22 20:48:30.533387 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.533261 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b0cd1b\"/\"default-dockercfg-9m5bv\"" Jul 22 20:48:30.533387 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.533278 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b0cd1b\"/\"openshift-service-ca.crt\"" Jul 22 20:48:30.533513 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.533310 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b0cd1b\"/\"rate-test-model-b0cd1b-kserve-self-signed-certs\"" Jul 22 20:48:30.538174 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.538147 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2"] Jul 22 20:48:30.617896 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.617860 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:48:30.618845 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.618823 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb56338-2087-4d7f-b988-0032faaa74a1-tls-certs\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.618948 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.618857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjr5s\" (UniqueName: \"kubernetes.io/projected/3fb56338-2087-4d7f-b988-0032faaa74a1-kube-api-access-mjr5s\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.618948 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.618891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-home\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.619066 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.618942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-tmp-dir\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.619066 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.618996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-dshm\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.619131 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.619083 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-model-cache\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.720472 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.720426 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-home\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.720472 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.720471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-tmp-dir\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.720730 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.720517 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-dshm\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.720730 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.720574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-model-cache\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.720730 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.720620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb56338-2087-4d7f-b988-0032faaa74a1-tls-certs\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.720730 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.720652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjr5s\" (UniqueName: \"kubernetes.io/projected/3fb56338-2087-4d7f-b988-0032faaa74a1-kube-api-access-mjr5s\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.720942 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.720885 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-home\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.720942 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.720902 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-model-cache\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.721110 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.721091 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-tmp-dir\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.722789 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.722767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-dshm\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.723054 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.723036 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb56338-2087-4d7f-b988-0032faaa74a1-tls-certs\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.728385 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.728363 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjr5s\" (UniqueName: \"kubernetes.io/projected/3fb56338-2087-4d7f-b988-0032faaa74a1-kube-api-access-mjr5s\") pod \"rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.841022 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.840884 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:30.971286 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:30.971256 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2"] Jul 22 20:48:30.973820 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:48:30.973792 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fb56338_2087_4d7f_b988_0032faaa74a1.slice/crio-417dce71c73730c339c7a24954a740aecc2ab5809cf911de5729d1536de96694 WatchSource:0}: Error finding container 417dce71c73730c339c7a24954a740aecc2ab5809cf911de5729d1536de96694: Status 404 returned error can't find the container with id 417dce71c73730c339c7a24954a740aecc2ab5809cf911de5729d1536de96694 Jul 22 20:48:31.056964 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:31.056924 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" event={"ID":"3fb56338-2087-4d7f-b988-0032faaa74a1","Type":"ContainerStarted","Data":"71b749aea557dfb68d8fe70bdf52d4bee269c6d5b625d07bb2a8a08515c3e8ab"} Jul 22 20:48:31.057136 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:31.056987 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" event={"ID":"3fb56338-2087-4d7f-b988-0032faaa74a1","Type":"ContainerStarted","Data":"417dce71c73730c339c7a24954a740aecc2ab5809cf911de5729d1536de96694"} Jul 22 20:48:31.074961 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:31.074909 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" podStartSLOduration=1.074893056 podStartE2EDuration="1.074893056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:48:31.072541489 +0000 UTC m=+2811.585421209" watchObservedRunningTime="2026-07-22 20:48:31.074893056 +0000 UTC m=+2811.587772763" Jul 22 20:48:34.369884 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:34.369847 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:48:37.650711 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.650650 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" podUID="a8780312-da91-4ae5-9f16-fec199c058d7" containerName="main" containerID="cri-o://4eed3bafd6e4db98b89cd8bde933ddb9919ba112d2ba00d20f27e2d99e082121" gracePeriod=45 Jul 22 20:48:37.902789 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.902723 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:48:37.991316 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.991281 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-home\") pod \"a8780312-da91-4ae5-9f16-fec199c058d7\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " Jul 22 20:48:37.991316 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.991322 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-model-cache\") pod \"a8780312-da91-4ae5-9f16-fec199c058d7\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " Jul 22 20:48:37.991564 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.991362 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-dshm\") pod \"a8780312-da91-4ae5-9f16-fec199c058d7\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " Jul 22 20:48:37.991564 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.991387 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkfhc\" (UniqueName: \"kubernetes.io/projected/a8780312-da91-4ae5-9f16-fec199c058d7-kube-api-access-kkfhc\") pod \"a8780312-da91-4ae5-9f16-fec199c058d7\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " Jul 22 20:48:37.991564 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.991435 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-tmp-dir\") pod \"a8780312-da91-4ae5-9f16-fec199c058d7\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " Jul 22 20:48:37.991564 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.991475 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8780312-da91-4ae5-9f16-fec199c058d7-tls-certs\") pod \"a8780312-da91-4ae5-9f16-fec199c058d7\" (UID: \"a8780312-da91-4ae5-9f16-fec199c058d7\") " Jul 22 20:48:37.991830 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.991624 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-home" (OuterVolumeSpecName: "home") pod "a8780312-da91-4ae5-9f16-fec199c058d7" (UID: "a8780312-da91-4ae5-9f16-fec199c058d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:48:37.991830 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.991637 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8780312-da91-4ae5-9f16-fec199c058d7" (UID: "a8780312-da91-4ae5-9f16-fec199c058d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:48:37.991830 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.991741 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8780312-da91-4ae5-9f16-fec199c058d7" (UID: "a8780312-da91-4ae5-9f16-fec199c058d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:48:37.991830 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.991767 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-home\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:48:37.991830 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.991788 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-model-cache\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:48:37.993564 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.993543 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-dshm" (OuterVolumeSpecName: "dshm") pod "a8780312-da91-4ae5-9f16-fec199c058d7" (UID: "a8780312-da91-4ae5-9f16-fec199c058d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:48:37.993649 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.993582 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8780312-da91-4ae5-9f16-fec199c058d7-kube-api-access-kkfhc" (OuterVolumeSpecName: "kube-api-access-kkfhc") pod "a8780312-da91-4ae5-9f16-fec199c058d7" (UID: "a8780312-da91-4ae5-9f16-fec199c058d7"). InnerVolumeSpecName "kube-api-access-kkfhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:48:37.993649 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:37.993603 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8780312-da91-4ae5-9f16-fec199c058d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8780312-da91-4ae5-9f16-fec199c058d7" (UID: "a8780312-da91-4ae5-9f16-fec199c058d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 20:48:38.087188 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:38.087151 2575 generic.go:356] "Generic (PLEG): container finished" podID="a8780312-da91-4ae5-9f16-fec199c058d7" containerID="4eed3bafd6e4db98b89cd8bde933ddb9919ba112d2ba00d20f27e2d99e082121" exitCode=0 Jul 22 20:48:38.087383 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:38.087199 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" Jul 22 20:48:38.087383 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:38.087205 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" event={"ID":"a8780312-da91-4ae5-9f16-fec199c058d7","Type":"ContainerDied","Data":"4eed3bafd6e4db98b89cd8bde933ddb9919ba112d2ba00d20f27e2d99e082121"} Jul 22 20:48:38.087383 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:38.087321 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8" event={"ID":"a8780312-da91-4ae5-9f16-fec199c058d7","Type":"ContainerDied","Data":"78012ccdfe84ced258fa32485dbb9ae041f3bb45e958b49336b54b9982b8c024"} Jul 22 20:48:38.087383 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:38.087344 2575 scope.go:117] "RemoveContainer" containerID="4eed3bafd6e4db98b89cd8bde933ddb9919ba112d2ba00d20f27e2d99e082121" Jul 22 20:48:38.092827 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:38.092799 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-tmp-dir\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:48:38.092827 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:38.092826 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8780312-da91-4ae5-9f16-fec199c058d7-tls-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:48:38.093101 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:38.092838 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8780312-da91-4ae5-9f16-fec199c058d7-dshm\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:48:38.093101 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:38.092849 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkfhc\" (UniqueName: \"kubernetes.io/projected/a8780312-da91-4ae5-9f16-fec199c058d7-kube-api-access-kkfhc\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:48:38.095826 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:38.095803 2575 scope.go:117] "RemoveContainer" containerID="4eed3bafd6e4db98b89cd8bde933ddb9919ba112d2ba00d20f27e2d99e082121" Jul 22 20:48:38.096128 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:48:38.096105 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eed3bafd6e4db98b89cd8bde933ddb9919ba112d2ba00d20f27e2d99e082121\": container with ID starting with 4eed3bafd6e4db98b89cd8bde933ddb9919ba112d2ba00d20f27e2d99e082121 not found: ID does not exist" containerID="4eed3bafd6e4db98b89cd8bde933ddb9919ba112d2ba00d20f27e2d99e082121" Jul 22 20:48:38.096223 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:38.096173 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eed3bafd6e4db98b89cd8bde933ddb9919ba112d2ba00d20f27e2d99e082121"} err="failed to get container status \"4eed3bafd6e4db98b89cd8bde933ddb9919ba112d2ba00d20f27e2d99e082121\": rpc error: code = NotFound desc = could not find container \"4eed3bafd6e4db98b89cd8bde933ddb9919ba112d2ba00d20f27e2d99e082121\": container with ID starting with 4eed3bafd6e4db98b89cd8bde933ddb9919ba112d2ba00d20f27e2d99e082121 not found: ID does not exist" Jul 22 20:48:38.102298 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:38.102276 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8"] Jul 22 20:48:38.106126 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:38.106103 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-198019/sub-test-model-198019-kserve-94b6869c5-trpm8"] Jul 22 20:48:40.007205 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:40.007170 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8780312-da91-4ae5-9f16-fec199c058d7" path="/var/lib/kubelet/pods/a8780312-da91-4ae5-9f16-fec199c058d7/volumes" Jul 22 20:48:40.841761 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:40.841723 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:40.841952 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:40.841784 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:40.854157 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:40.854132 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:41.118447 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:41.118364 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:48:59.552931 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:48:59.552893 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:49:03.472077 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:03.472038 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:49:22.064052 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:22.064013 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:49:32.854665 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:32.854610 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:49:34.197741 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:34.197707 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:49:38.174996 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:38.174947 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:49:40.237969 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:40.237939 2575 scope.go:117] "RemoveContainer" containerID="1fef0798aeaafcd456eacabb7ac269264f63fb2774827c38221dd5ac5b07b2c5" Jul 22 20:49:40.275955 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:40.275916 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:49:40.377003 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:40.376944 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:49:44.653546 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:44.653511 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2"] Jul 22 20:49:44.672784 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:44.672752 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:49:45.550738 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:45.550705 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:49:57.185619 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:57.185577 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:49:59.673571 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:59.673509 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" podUID="3fb56338-2087-4d7f-b988-0032faaa74a1" containerName="main" containerID="cri-o://71b749aea557dfb68d8fe70bdf52d4bee269c6d5b625d07bb2a8a08515c3e8ab" gracePeriod=45 Jul 22 20:49:59.922300 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:49:59.922274 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:50:00.113605 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.113547 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjr5s\" (UniqueName: \"kubernetes.io/projected/3fb56338-2087-4d7f-b988-0032faaa74a1-kube-api-access-mjr5s\") pod \"3fb56338-2087-4d7f-b988-0032faaa74a1\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " Jul 22 20:50:00.113605 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.113620 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-dshm\") pod \"3fb56338-2087-4d7f-b988-0032faaa74a1\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " Jul 22 20:50:00.113901 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.113650 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-model-cache\") pod \"3fb56338-2087-4d7f-b988-0032faaa74a1\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " Jul 22 20:50:00.113901 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.113711 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb56338-2087-4d7f-b988-0032faaa74a1-tls-certs\") pod \"3fb56338-2087-4d7f-b988-0032faaa74a1\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " Jul 22 20:50:00.113901 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.113733 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-home\") pod \"3fb56338-2087-4d7f-b988-0032faaa74a1\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " Jul 22 20:50:00.113901 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.113791 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-tmp-dir\") pod \"3fb56338-2087-4d7f-b988-0032faaa74a1\" (UID: \"3fb56338-2087-4d7f-b988-0032faaa74a1\") " Jul 22 20:50:00.114139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.114016 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "3fb56338-2087-4d7f-b988-0032faaa74a1" (UID: "3fb56338-2087-4d7f-b988-0032faaa74a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:00.114139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.114070 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-home" (OuterVolumeSpecName: "home") pod "3fb56338-2087-4d7f-b988-0032faaa74a1" (UID: "3fb56338-2087-4d7f-b988-0032faaa74a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:00.114265 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.114156 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-model-cache\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:00.114265 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.114152 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3fb56338-2087-4d7f-b988-0032faaa74a1" (UID: "3fb56338-2087-4d7f-b988-0032faaa74a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:00.116016 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.115991 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb56338-2087-4d7f-b988-0032faaa74a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fb56338-2087-4d7f-b988-0032faaa74a1" (UID: "3fb56338-2087-4d7f-b988-0032faaa74a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 20:50:00.116305 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.116282 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-dshm" (OuterVolumeSpecName: "dshm") pod "3fb56338-2087-4d7f-b988-0032faaa74a1" (UID: "3fb56338-2087-4d7f-b988-0032faaa74a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:00.116367 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.116285 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb56338-2087-4d7f-b988-0032faaa74a1-kube-api-access-mjr5s" (OuterVolumeSpecName: "kube-api-access-mjr5s") pod "3fb56338-2087-4d7f-b988-0032faaa74a1" (UID: "3fb56338-2087-4d7f-b988-0032faaa74a1"). InnerVolumeSpecName "kube-api-access-mjr5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:50:00.214833 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.214793 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb56338-2087-4d7f-b988-0032faaa74a1-tls-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:00.214833 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.214825 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-home\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:00.214833 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.214834 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-tmp-dir\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:00.214833 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.214845 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjr5s\" (UniqueName: \"kubernetes.io/projected/3fb56338-2087-4d7f-b988-0032faaa74a1-kube-api-access-mjr5s\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:00.215154 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.214853 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb56338-2087-4d7f-b988-0032faaa74a1-dshm\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:00.433964 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.433870 2575 generic.go:356] "Generic (PLEG): container finished" podID="3fb56338-2087-4d7f-b988-0032faaa74a1" containerID="71b749aea557dfb68d8fe70bdf52d4bee269c6d5b625d07bb2a8a08515c3e8ab" exitCode=0 Jul 22 20:50:00.433964 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.433914 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" Jul 22 20:50:00.433964 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.433953 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" event={"ID":"3fb56338-2087-4d7f-b988-0032faaa74a1","Type":"ContainerDied","Data":"71b749aea557dfb68d8fe70bdf52d4bee269c6d5b625d07bb2a8a08515c3e8ab"} Jul 22 20:50:00.434231 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.434016 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2" event={"ID":"3fb56338-2087-4d7f-b988-0032faaa74a1","Type":"ContainerDied","Data":"417dce71c73730c339c7a24954a740aecc2ab5809cf911de5729d1536de96694"} Jul 22 20:50:00.434231 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.434040 2575 scope.go:117] "RemoveContainer" containerID="71b749aea557dfb68d8fe70bdf52d4bee269c6d5b625d07bb2a8a08515c3e8ab" Jul 22 20:50:00.442835 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.442814 2575 scope.go:117] "RemoveContainer" containerID="71b749aea557dfb68d8fe70bdf52d4bee269c6d5b625d07bb2a8a08515c3e8ab" Jul 22 20:50:00.443111 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:50:00.443093 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b749aea557dfb68d8fe70bdf52d4bee269c6d5b625d07bb2a8a08515c3e8ab\": container with ID starting with 71b749aea557dfb68d8fe70bdf52d4bee269c6d5b625d07bb2a8a08515c3e8ab not found: ID does not exist" containerID="71b749aea557dfb68d8fe70bdf52d4bee269c6d5b625d07bb2a8a08515c3e8ab" Jul 22 20:50:00.443176 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.443121 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b749aea557dfb68d8fe70bdf52d4bee269c6d5b625d07bb2a8a08515c3e8ab"} err="failed to get container status \"71b749aea557dfb68d8fe70bdf52d4bee269c6d5b625d07bb2a8a08515c3e8ab\": rpc error: code = NotFound desc = could not find container \"71b749aea557dfb68d8fe70bdf52d4bee269c6d5b625d07bb2a8a08515c3e8ab\": container with ID starting with 71b749aea557dfb68d8fe70bdf52d4bee269c6d5b625d07bb2a8a08515c3e8ab not found: ID does not exist" Jul 22 20:50:00.455692 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.455664 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2"] Jul 22 20:50:00.458178 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:00.458156 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b0cd1b/rate-test-model-b0cd1b-kserve-68d797b8b7-4lxq2"] Jul 22 20:50:02.007021 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:02.006968 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fb56338-2087-4d7f-b988-0032faaa74a1" path="/var/lib/kubelet/pods/3fb56338-2087-4d7f-b988-0032faaa74a1/volumes" Jul 22 20:50:05.333722 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.333677 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv"] Jul 22 20:50:05.334360 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.333991 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fb56338-2087-4d7f-b988-0032faaa74a1" containerName="main" Jul 22 20:50:05.334360 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.334003 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fb56338-2087-4d7f-b988-0032faaa74a1" containerName="main" Jul 22 20:50:05.334360 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.334012 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8780312-da91-4ae5-9f16-fec199c058d7" containerName="main" Jul 22 20:50:05.334360 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.334018 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8780312-da91-4ae5-9f16-fec199c058d7" containerName="main" Jul 22 20:50:05.334360 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.334063 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8780312-da91-4ae5-9f16-fec199c058d7" containerName="main" Jul 22 20:50:05.334360 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.334073 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fb56338-2087-4d7f-b988-0032faaa74a1" containerName="main" Jul 22 20:50:05.338127 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.338103 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv" Jul 22 20:50:05.340632 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.340603 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 20:50:05.340632 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.340603 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjqt7\"" Jul 22 20:50:05.343546 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.343517 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv"] Jul 22 20:50:05.361649 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.361619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56578423-87a3-4db3-95b9-732ddcc50356-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv\" (UID: \"56578423-87a3-4db3-95b9-732ddcc50356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv" Jul 22 20:50:05.462735 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.462700 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56578423-87a3-4db3-95b9-732ddcc50356-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv\" (UID: \"56578423-87a3-4db3-95b9-732ddcc50356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv" Jul 22 20:50:05.463406 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.463385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56578423-87a3-4db3-95b9-732ddcc50356-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv\" (UID: \"56578423-87a3-4db3-95b9-732ddcc50356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv" Jul 22 20:50:05.649241 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.649147 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv" Jul 22 20:50:05.778467 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.778440 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv"] Jul 22 20:50:05.780870 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:50:05.780842 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56578423_87a3_4db3_95b9_732ddcc50356.slice/crio-1c33c2a77c85ebef75d73f5b3e1eb3d3d53eb57078850ce96f70447a9126c53f WatchSource:0}: Error finding container 1c33c2a77c85ebef75d73f5b3e1eb3d3d53eb57078850ce96f70447a9126c53f: Status 404 returned error can't find the container with id 1c33c2a77c85ebef75d73f5b3e1eb3d3d53eb57078850ce96f70447a9126c53f Jul 22 20:50:05.844921 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.844890 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:05.951438 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:05.951402 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:06.462469 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:06.462422 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv" event={"ID":"56578423-87a3-4db3-95b9-732ddcc50356","Type":"ContainerStarted","Data":"5457a6523e02c0e59f09a4489069280e8f8aa21bab82662c65437d87a20319fb"} Jul 22 20:50:06.462862 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:06.462488 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv" event={"ID":"56578423-87a3-4db3-95b9-732ddcc50356","Type":"ContainerStarted","Data":"1c33c2a77c85ebef75d73f5b3e1eb3d3d53eb57078850ce96f70447a9126c53f"} Jul 22 20:50:06.482918 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:06.482862 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv" podStartSLOduration=1.482846334 podStartE2EDuration="1.482846334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:50:06.48043596 +0000 UTC m=+2906.993315667" watchObservedRunningTime="2026-07-22 20:50:06.482846334 +0000 UTC m=+2906.995726041" Jul 22 20:50:07.467661 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:07.467624 2575 generic.go:356] "Generic (PLEG): container finished" podID="56578423-87a3-4db3-95b9-732ddcc50356" containerID="5457a6523e02c0e59f09a4489069280e8f8aa21bab82662c65437d87a20319fb" exitCode=0 Jul 22 20:50:07.468058 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:07.467707 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv" event={"ID":"56578423-87a3-4db3-95b9-732ddcc50356","Type":"ContainerDied","Data":"5457a6523e02c0e59f09a4489069280e8f8aa21bab82662c65437d87a20319fb"} Jul 22 20:50:08.087929 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:08.087882 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:08.181771 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:08.181739 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:08.607378 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:08.607351 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv" Jul 22 20:50:08.690336 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:08.690299 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56578423-87a3-4db3-95b9-732ddcc50356-maas-api-service-ca\") pod \"56578423-87a3-4db3-95b9-732ddcc50356\" (UID: \"56578423-87a3-4db3-95b9-732ddcc50356\") " Jul 22 20:50:08.690699 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:08.690675 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56578423-87a3-4db3-95b9-732ddcc50356-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "56578423-87a3-4db3-95b9-732ddcc50356" (UID: "56578423-87a3-4db3-95b9-732ddcc50356"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:50:08.791743 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:08.791657 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56578423-87a3-4db3-95b9-732ddcc50356-maas-api-service-ca\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:09.477613 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:09.477581 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv" event={"ID":"56578423-87a3-4db3-95b9-732ddcc50356","Type":"ContainerDied","Data":"1c33c2a77c85ebef75d73f5b3e1eb3d3d53eb57078850ce96f70447a9126c53f"} Jul 22 20:50:09.477786 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:09.477633 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c33c2a77c85ebef75d73f5b3e1eb3d3d53eb57078850ce96f70447a9126c53f" Jul 22 20:50:09.477786 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:09.477593 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv" Jul 22 20:50:11.596507 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:11.596446 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:11.774491 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:11.774452 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:15.932655 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:15.932613 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:16.764107 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:16.764073 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:16.871107 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:16.871080 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:19.272654 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:19.272619 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:50:20.432641 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:20.432607 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:20.455101 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:20.455060 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv"] Jul 22 20:50:20.465172 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:20.465136 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b0cd1b-2x4hv"] Jul 22 20:50:20.831893 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:20.831856 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:22.008929 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:22.008894 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56578423-87a3-4db3-95b9-732ddcc50356" path="/var/lib/kubelet/pods/56578423-87a3-4db3-95b9-732ddcc50356/volumes" Jul 22 20:50:26.642016 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:26.641932 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:26.746464 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:26.746432 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:26.894398 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:26.894305 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps"] Jul 22 20:50:26.894838 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:26.894806 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerName="istio-proxy" containerID="cri-o://fcb095dc94b2289428f84ce4363685787b2c7e4bc393020431acb50c00ac72c5" gracePeriod=30 Jul 22 20:50:26.921956 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:26.921915 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:27.173385 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:27.173297 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:27.818087 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:27.818053 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:27.922259 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:27.922226 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:31.027736 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:31.027693 2575 patch_prober.go:28] interesting pod/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 20:50:31.028141 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:31.027752 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 20:50:32.141428 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.141399 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:50:32.297854 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.297809 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-credential-socket\") pod \"c024094b-887e-4fc9-9a2c-abadc1db217c\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " Jul 22 20:50:32.297854 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.297867 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-data\") pod \"c024094b-887e-4fc9-9a2c-abadc1db217c\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " Jul 22 20:50:32.298139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.297894 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-token\") pod \"c024094b-887e-4fc9-9a2c-abadc1db217c\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " Jul 22 20:50:32.298139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.297915 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-envoy\") pod \"c024094b-887e-4fc9-9a2c-abadc1db217c\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " Jul 22 20:50:32.298139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.297943 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-workload-socket\") pod \"c024094b-887e-4fc9-9a2c-abadc1db217c\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " Jul 22 20:50:32.298139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.297972 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c024094b-887e-4fc9-9a2c-abadc1db217c-istiod-ca-cert\") pod \"c024094b-887e-4fc9-9a2c-abadc1db217c\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " Jul 22 20:50:32.298139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.298033 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-podinfo\") pod \"c024094b-887e-4fc9-9a2c-abadc1db217c\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " Jul 22 20:50:32.298139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.298052 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-workload-certs\") pod \"c024094b-887e-4fc9-9a2c-abadc1db217c\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " Jul 22 20:50:32.298139 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.298077 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9blq\" (UniqueName: \"kubernetes.io/projected/c024094b-887e-4fc9-9a2c-abadc1db217c-kube-api-access-w9blq\") pod \"c024094b-887e-4fc9-9a2c-abadc1db217c\" (UID: \"c024094b-887e-4fc9-9a2c-abadc1db217c\") " Jul 22 20:50:32.298489 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.298175 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c024094b-887e-4fc9-9a2c-abadc1db217c" (UID: "c024094b-887e-4fc9-9a2c-abadc1db217c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:32.298489 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.298303 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c024094b-887e-4fc9-9a2c-abadc1db217c" (UID: "c024094b-887e-4fc9-9a2c-abadc1db217c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:32.298489 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.298321 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-credential-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:32.298646 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.298506 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c024094b-887e-4fc9-9a2c-abadc1db217c" (UID: "c024094b-887e-4fc9-9a2c-abadc1db217c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:32.298646 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.298593 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c024094b-887e-4fc9-9a2c-abadc1db217c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c024094b-887e-4fc9-9a2c-abadc1db217c" (UID: "c024094b-887e-4fc9-9a2c-abadc1db217c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:50:32.298851 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.298824 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-data" (OuterVolumeSpecName: "istio-data") pod "c024094b-887e-4fc9-9a2c-abadc1db217c" (UID: "c024094b-887e-4fc9-9a2c-abadc1db217c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:32.300333 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.300303 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c024094b-887e-4fc9-9a2c-abadc1db217c" (UID: "c024094b-887e-4fc9-9a2c-abadc1db217c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:32.300424 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.300380 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c024094b-887e-4fc9-9a2c-abadc1db217c-kube-api-access-w9blq" (OuterVolumeSpecName: "kube-api-access-w9blq") pod "c024094b-887e-4fc9-9a2c-abadc1db217c" (UID: "c024094b-887e-4fc9-9a2c-abadc1db217c"). InnerVolumeSpecName "kube-api-access-w9blq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:50:32.300700 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.300681 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c024094b-887e-4fc9-9a2c-abadc1db217c" (UID: "c024094b-887e-4fc9-9a2c-abadc1db217c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 20:50:32.300788 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.300700 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-token" (OuterVolumeSpecName: "istio-token") pod "c024094b-887e-4fc9-9a2c-abadc1db217c" (UID: "c024094b-887e-4fc9-9a2c-abadc1db217c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:50:32.399718 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.399666 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c024094b-887e-4fc9-9a2c-abadc1db217c-istiod-ca-cert\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:32.399718 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.399712 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-podinfo\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:32.399718 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.399721 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-workload-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:32.399718 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.399731 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9blq\" (UniqueName: \"kubernetes.io/projected/c024094b-887e-4fc9-9a2c-abadc1db217c-kube-api-access-w9blq\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:32.399718 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.399741 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-data\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:32.400064 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.399749 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-token\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:32.400064 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.399757 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-istio-envoy\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:32.400064 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.399766 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c024094b-887e-4fc9-9a2c-abadc1db217c-workload-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:32.572203 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.572106 2575 generic.go:356] "Generic (PLEG): container finished" podID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerID="fcb095dc94b2289428f84ce4363685787b2c7e4bc393020431acb50c00ac72c5" exitCode=0 Jul 22 20:50:32.572203 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.572164 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" Jul 22 20:50:32.572203 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.572190 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" event={"ID":"c024094b-887e-4fc9-9a2c-abadc1db217c","Type":"ContainerDied","Data":"fcb095dc94b2289428f84ce4363685787b2c7e4bc393020431acb50c00ac72c5"} Jul 22 20:50:32.572477 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.572246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps" event={"ID":"c024094b-887e-4fc9-9a2c-abadc1db217c","Type":"ContainerDied","Data":"4f75a3ab2159cccc12a9b9143b79326e7c23b69e5449257241788795b6fe86fb"} Jul 22 20:50:32.572477 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.572271 2575 scope.go:117] "RemoveContainer" containerID="fcb095dc94b2289428f84ce4363685787b2c7e4bc393020431acb50c00ac72c5" Jul 22 20:50:32.580948 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.580932 2575 scope.go:117] "RemoveContainer" containerID="520c2cd218e99c1a89eb2f093cf993d3bc03f5087ed50de939ba70277b09cda9" Jul 22 20:50:32.588610 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.588582 2575 scope.go:117] "RemoveContainer" containerID="fcb095dc94b2289428f84ce4363685787b2c7e4bc393020431acb50c00ac72c5" Jul 22 20:50:32.588856 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:50:32.588836 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb095dc94b2289428f84ce4363685787b2c7e4bc393020431acb50c00ac72c5\": container with ID starting with fcb095dc94b2289428f84ce4363685787b2c7e4bc393020431acb50c00ac72c5 not found: ID does not exist" containerID="fcb095dc94b2289428f84ce4363685787b2c7e4bc393020431acb50c00ac72c5" Jul 22 20:50:32.588904 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.588866 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb095dc94b2289428f84ce4363685787b2c7e4bc393020431acb50c00ac72c5"} err="failed to get container status \"fcb095dc94b2289428f84ce4363685787b2c7e4bc393020431acb50c00ac72c5\": rpc error: code = NotFound desc = could not find container \"fcb095dc94b2289428f84ce4363685787b2c7e4bc393020431acb50c00ac72c5\": container with ID starting with fcb095dc94b2289428f84ce4363685787b2c7e4bc393020431acb50c00ac72c5 not found: ID does not exist" Jul 22 20:50:32.588904 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.588885 2575 scope.go:117] "RemoveContainer" containerID="520c2cd218e99c1a89eb2f093cf993d3bc03f5087ed50de939ba70277b09cda9" Jul 22 20:50:32.589168 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:50:32.589142 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520c2cd218e99c1a89eb2f093cf993d3bc03f5087ed50de939ba70277b09cda9\": container with ID starting with 520c2cd218e99c1a89eb2f093cf993d3bc03f5087ed50de939ba70277b09cda9 not found: ID does not exist" containerID="520c2cd218e99c1a89eb2f093cf993d3bc03f5087ed50de939ba70277b09cda9" Jul 22 20:50:32.589222 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.589172 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520c2cd218e99c1a89eb2f093cf993d3bc03f5087ed50de939ba70277b09cda9"} err="failed to get container status \"520c2cd218e99c1a89eb2f093cf993d3bc03f5087ed50de939ba70277b09cda9\": rpc error: code = NotFound desc = could not find container \"520c2cd218e99c1a89eb2f093cf993d3bc03f5087ed50de939ba70277b09cda9\": container with ID starting with 520c2cd218e99c1a89eb2f093cf993d3bc03f5087ed50de939ba70277b09cda9 not found: ID does not exist" Jul 22 20:50:32.596600 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.596575 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps"] Jul 22 20:50:32.600949 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:32.600922 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-b0cd1b-openshift-default-68cfb649cf-ctgps"] Jul 22 20:50:34.006872 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:34.006838 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" path="/var/lib/kubelet/pods/c024094b-887e-4fc9-9a2c-abadc1db217c/volumes" Jul 22 20:50:37.978396 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:37.978351 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:37.979044 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:37.979023 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-198019-29745885-h7cc5"] Jul 22 20:50:37.981590 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:37.981568 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-198019-29745885-h7cc5"] Jul 22 20:50:38.006920 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:38.006886 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4ab4cdd-f494-4a7e-949e-0d848fb3349a" path="/var/lib/kubelet/pods/e4ab4cdd-f494-4a7e-949e-0d848fb3349a/volumes" Jul 22 20:50:40.872291 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:40.872257 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcbb2"] Jul 22 20:50:42.372703 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:42.372664 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:42.764329 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:42.764293 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:49.082523 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:49.082487 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:49.153387 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:49.153354 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:49.352493 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:49.352407 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd"] Jul 22 20:50:49.352643 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:49.352617 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerName="istio-proxy" containerID="cri-o://b69d80e97f1e0717d9efe884f7298b4334ee49944175b839bd0c2dd82e774fd8" gracePeriod=30 Jul 22 20:50:49.369433 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:49.369401 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:49.620453 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:49.620362 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:50:53.333662 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:53.333628 2575 patch_prober.go:28] interesting pod/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 20:50:53.334048 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:53.333687 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 20:50:54.595015 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.594988 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:50:54.664025 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.663897 2575 generic.go:356] "Generic (PLEG): container finished" podID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerID="b69d80e97f1e0717d9efe884f7298b4334ee49944175b839bd0c2dd82e774fd8" exitCode=0 Jul 22 20:50:54.664025 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.663954 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" Jul 22 20:50:54.664025 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.664009 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" event={"ID":"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1","Type":"ContainerDied","Data":"b69d80e97f1e0717d9efe884f7298b4334ee49944175b839bd0c2dd82e774fd8"} Jul 22 20:50:54.664315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.664066 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd" event={"ID":"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1","Type":"ContainerDied","Data":"5889c392d188cdb3b3d2ea6851acd5397d1985761d9d65b5997651c8e0a660f8"} Jul 22 20:50:54.664315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.664089 2575 scope.go:117] "RemoveContainer" containerID="b69d80e97f1e0717d9efe884f7298b4334ee49944175b839bd0c2dd82e774fd8" Jul 22 20:50:54.672310 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.672290 2575 scope.go:117] "RemoveContainer" containerID="963c72e76d3e820489bb6a8fe9adafd450870233fecbb0ce82bf59aabb4e1a28" Jul 22 20:50:54.681721 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.681693 2575 scope.go:117] "RemoveContainer" containerID="b69d80e97f1e0717d9efe884f7298b4334ee49944175b839bd0c2dd82e774fd8" Jul 22 20:50:54.682191 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:50:54.682169 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69d80e97f1e0717d9efe884f7298b4334ee49944175b839bd0c2dd82e774fd8\": container with ID starting with b69d80e97f1e0717d9efe884f7298b4334ee49944175b839bd0c2dd82e774fd8 not found: ID does not exist" containerID="b69d80e97f1e0717d9efe884f7298b4334ee49944175b839bd0c2dd82e774fd8" Jul 22 20:50:54.682334 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.682205 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69d80e97f1e0717d9efe884f7298b4334ee49944175b839bd0c2dd82e774fd8"} err="failed to get container status \"b69d80e97f1e0717d9efe884f7298b4334ee49944175b839bd0c2dd82e774fd8\": rpc error: code = NotFound desc = could not find container \"b69d80e97f1e0717d9efe884f7298b4334ee49944175b839bd0c2dd82e774fd8\": container with ID starting with b69d80e97f1e0717d9efe884f7298b4334ee49944175b839bd0c2dd82e774fd8 not found: ID does not exist" Jul 22 20:50:54.682334 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.682231 2575 scope.go:117] "RemoveContainer" containerID="963c72e76d3e820489bb6a8fe9adafd450870233fecbb0ce82bf59aabb4e1a28" Jul 22 20:50:54.682499 ip-10-0-143-196 kubenswrapper[2575]: E0722 20:50:54.682479 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963c72e76d3e820489bb6a8fe9adafd450870233fecbb0ce82bf59aabb4e1a28\": container with ID starting with 963c72e76d3e820489bb6a8fe9adafd450870233fecbb0ce82bf59aabb4e1a28 not found: ID does not exist" containerID="963c72e76d3e820489bb6a8fe9adafd450870233fecbb0ce82bf59aabb4e1a28" Jul 22 20:50:54.682540 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.682509 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963c72e76d3e820489bb6a8fe9adafd450870233fecbb0ce82bf59aabb4e1a28"} err="failed to get container status \"963c72e76d3e820489bb6a8fe9adafd450870233fecbb0ce82bf59aabb4e1a28\": rpc error: code = NotFound desc = could not find container \"963c72e76d3e820489bb6a8fe9adafd450870233fecbb0ce82bf59aabb4e1a28\": container with ID starting with 963c72e76d3e820489bb6a8fe9adafd450870233fecbb0ce82bf59aabb4e1a28 not found: ID does not exist" Jul 22 20:50:54.698921 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.698893 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-workload-certs\") pod \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " Jul 22 20:50:54.699080 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.698928 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-podinfo\") pod \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " Jul 22 20:50:54.699080 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.698968 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-envoy\") pod \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " Jul 22 20:50:54.699080 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.699008 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-data\") pod \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " Jul 22 20:50:54.699080 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.699041 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppw8k\" (UniqueName: \"kubernetes.io/projected/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-kube-api-access-ppw8k\") pod \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " Jul 22 20:50:54.699291 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.699087 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istiod-ca-cert\") pod \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " Jul 22 20:50:54.699291 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.699123 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-credential-socket\") pod \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " Jul 22 20:50:54.699291 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.699149 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-workload-socket\") pod \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " Jul 22 20:50:54.699291 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.699175 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-token\") pod \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\" (UID: \"e63ce6b8-1453-41f0-a6a5-7ed73732e3b1\") " Jul 22 20:50:54.699291 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.699240 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" (UID: "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:54.699543 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.699368 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" (UID: "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:54.699543 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.699502 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-credential-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:54.699543 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.699523 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-workload-certs\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:54.699543 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.699524 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" (UID: "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:54.699743 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.699565 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" (UID: "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 20:50:54.700125 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.700086 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" (UID: "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:54.701315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.701279 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" (UID: "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 20:50:54.701315 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.701293 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" (UID: "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 20:50:54.701456 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.701326 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-kube-api-access-ppw8k" (OuterVolumeSpecName: "kube-api-access-ppw8k") pod "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" (UID: "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1"). InnerVolumeSpecName "kube-api-access-ppw8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:50:54.701456 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.701399 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" (UID: "e63ce6b8-1453-41f0-a6a5-7ed73732e3b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 20:50:54.800809 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.800766 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-workload-socket\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:54.800809 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.800797 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-token\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:54.800809 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.800807 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-podinfo\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:54.800809 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.800815 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-envoy\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:54.800809 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.800824 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istio-data\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:54.801146 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.800832 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppw8k\" (UniqueName: \"kubernetes.io/projected/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-kube-api-access-ppw8k\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:54.801146 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.800841 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1-istiod-ca-cert\") on node \"ip-10-0-143-196.ec2.internal\" DevicePath \"\"" Jul 22 20:50:54.986330 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.986294 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd"] Jul 22 20:50:54.990647 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:54.990620 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-198019-openshift-default-85568fb999-qdmpd"] Jul 22 20:50:55.816255 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:55.816216 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64cc9776b5-sqx22_e2d423d9-17cc-4c4a-b492-3416c04b332c/manager/0.log" Jul 22 20:50:56.006919 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:56.006884 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" path="/var/lib/kubelet/pods/e63ce6b8-1453-41f0-a6a5-7ed73732e3b1/volumes" Jul 22 20:50:56.360591 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:56.360548 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9fc56f4f-78xpl_97c53fcf-3ae1-49af-a341-42aab4dfba86/copy-manifests/0.log" Jul 22 20:50:56.368613 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:56.368586 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9fc56f4f-78xpl_97c53fcf-3ae1-49af-a341-42aab4dfba86/manager/0.log" Jul 22 20:50:57.072439 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:57.072408 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745870-m4zxk_3b0e23b8-3ec2-43a8-937c-ee4da5f6a63b/cleanup/0.log" Jul 22 20:50:59.361647 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:59.361616 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fcbb2_28678c94-d0ce-4e87-8336-da8bb10eb673/limitador/0.log" Jul 22 20:50:59.824795 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:59.824764 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46482l_cf0c6221-5afb-4a54-8ee5-c856d3a0e77d/istio-proxy/0.log" Jul 22 20:50:59.948811 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:50:59.948776 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5nc2r_cb12e370-8f88-49af-8643-ccffb2f996a6/discovery/0.log" Jul 22 20:51:07.644934 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.644897 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nzwv7/must-gather-dcvck"] Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645173 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerName="istio-proxy" Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645184 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerName="istio-proxy" Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645202 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerName="istio-proxy" Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645207 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerName="istio-proxy" Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645213 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerName="istio-proxy" Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645218 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerName="istio-proxy" Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645226 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerName="istio-proxy" Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645231 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerName="istio-proxy" Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645239 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56578423-87a3-4db3-95b9-732ddcc50356" containerName="revoke-keys" Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645245 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56578423-87a3-4db3-95b9-732ddcc50356" containerName="revoke-keys" Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645285 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerName="istio-proxy" Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645294 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerName="istio-proxy" Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645300 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e63ce6b8-1453-41f0-a6a5-7ed73732e3b1" containerName="istio-proxy" Jul 22 20:51:07.645326 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645310 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="56578423-87a3-4db3-95b9-732ddcc50356" containerName="revoke-keys" Jul 22 20:51:07.645753 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.645418 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c024094b-887e-4fc9-9a2c-abadc1db217c" containerName="istio-proxy" Jul 22 20:51:07.649771 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.649749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzwv7/must-gather-dcvck" Jul 22 20:51:07.652118 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.652097 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nzwv7\"/\"default-dockercfg-mc8gv\"" Jul 22 20:51:07.652273 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.652253 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nzwv7\"/\"openshift-service-ca.crt\"" Jul 22 20:51:07.654547 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.654530 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nzwv7\"/\"kube-root-ca.crt\"" Jul 22 20:51:07.659220 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.659195 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzwv7/must-gather-dcvck"] Jul 22 20:51:07.670884 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.670856 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:51:07.704557 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.704513 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/299ed99c-89e8-4e73-833d-b31b6a487fa0-must-gather-output\") pod \"must-gather-dcvck\" (UID: \"299ed99c-89e8-4e73-833d-b31b6a487fa0\") " pod="openshift-must-gather-nzwv7/must-gather-dcvck" Jul 22 20:51:07.704725 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.704640 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x9z2\" (UniqueName: \"kubernetes.io/projected/299ed99c-89e8-4e73-833d-b31b6a487fa0-kube-api-access-2x9z2\") pod \"must-gather-dcvck\" (UID: \"299ed99c-89e8-4e73-833d-b31b6a487fa0\") " pod="openshift-must-gather-nzwv7/must-gather-dcvck" Jul 22 20:51:07.765611 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.765577 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" limit=1024 Jul 22 20:51:07.805894 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.805862 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x9z2\" (UniqueName: \"kubernetes.io/projected/299ed99c-89e8-4e73-833d-b31b6a487fa0-kube-api-access-2x9z2\") pod \"must-gather-dcvck\" (UID: \"299ed99c-89e8-4e73-833d-b31b6a487fa0\") " pod="openshift-must-gather-nzwv7/must-gather-dcvck" Jul 22 20:51:07.806109 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.805928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/299ed99c-89e8-4e73-833d-b31b6a487fa0-must-gather-output\") pod \"must-gather-dcvck\" (UID: \"299ed99c-89e8-4e73-833d-b31b6a487fa0\") " pod="openshift-must-gather-nzwv7/must-gather-dcvck" Jul 22 20:51:07.806302 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.806282 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/299ed99c-89e8-4e73-833d-b31b6a487fa0-must-gather-output\") pod \"must-gather-dcvck\" (UID: \"299ed99c-89e8-4e73-833d-b31b6a487fa0\") " pod="openshift-must-gather-nzwv7/must-gather-dcvck" Jul 22 20:51:07.814457 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.814430 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x9z2\" (UniqueName: \"kubernetes.io/projected/299ed99c-89e8-4e73-833d-b31b6a487fa0-kube-api-access-2x9z2\") pod \"must-gather-dcvck\" (UID: \"299ed99c-89e8-4e73-833d-b31b6a487fa0\") " pod="openshift-must-gather-nzwv7/must-gather-dcvck" Jul 22 20:51:07.958772 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:07.958738 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzwv7/must-gather-dcvck" Jul 22 20:51:08.089221 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:08.089181 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzwv7/must-gather-dcvck"] Jul 22 20:51:08.092654 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:51:08.092629 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod299ed99c_89e8_4e73_833d_b31b6a487fa0.slice/crio-efa3efdf87540b1c3c591710472452f66c1da4913a678627e6c5ae82b1943555 WatchSource:0}: Error finding container efa3efdf87540b1c3c591710472452f66c1da4913a678627e6c5ae82b1943555: Status 404 returned error can't find the container with id efa3efdf87540b1c3c591710472452f66c1da4913a678627e6c5ae82b1943555 Jul 22 20:51:08.094481 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:08.094466 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 20:51:08.724824 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:08.724780 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzwv7/must-gather-dcvck" event={"ID":"299ed99c-89e8-4e73-833d-b31b6a487fa0","Type":"ContainerStarted","Data":"efa3efdf87540b1c3c591710472452f66c1da4913a678627e6c5ae82b1943555"} Jul 22 20:51:09.733203 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:09.732638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzwv7/must-gather-dcvck" event={"ID":"299ed99c-89e8-4e73-833d-b31b6a487fa0","Type":"ContainerStarted","Data":"b819aab3f2afd21d1d441e14365b1d5b39a7e9c3339c4fb82f849e0839d6ff4c"} Jul 22 20:51:09.733203 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:09.732692 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzwv7/must-gather-dcvck" event={"ID":"299ed99c-89e8-4e73-833d-b31b6a487fa0","Type":"ContainerStarted","Data":"21e6d2cdd18aaf65bbbc6c434a8032bb0124a89540b0147322be5eaa49f0909b"} Jul 22 20:51:09.750106 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:09.750044 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nzwv7/must-gather-dcvck" podStartSLOduration=1.466718906 podStartE2EDuration="2.75002302s" totalImagesPullingTime="1.283304114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 20:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:51:09.748376891 +0000 UTC m=+2970.261256669" watchObservedRunningTime="2026-07-22 20:51:09.75002302 +0000 UTC m=+2970.262902734" Jul 22 20:51:10.906678 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:10.906644 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j7h4z_67099d8a-f2c3-492f-a99f-6d81f030bc54/global-pull-secret-syncer/0.log" Jul 22 20:51:11.051521 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:11.051476 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7jg4v_90fd0be7-5de1-41c4-aeb0-59685a8907c1/konnectivity-agent/0.log" Jul 22 20:51:11.204775 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:11.204733 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-196.ec2.internal_f16d2aa87b61c01543e0be055f07431f/haproxy/0.log" Jul 22 20:51:17.261235 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:17.261199 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fcbb2_28678c94-d0ce-4e87-8336-da8bb10eb673/limitador/0.log" Jul 22 20:51:19.317091 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:19.317057 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f7f6d6f69-x4vql_431ab348-975c-4e0d-9a5a-d45eaf8f92df/metrics-server/0.log" Jul 22 20:51:19.369591 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:19.369556 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6q5qz_de9b3c28-175e-47ef-9138-4f455e701503/node-exporter/0.log" Jul 22 20:51:19.406617 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:19.406581 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6q5qz_de9b3c28-175e-47ef-9138-4f455e701503/kube-rbac-proxy/0.log" Jul 22 20:51:19.429408 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:19.429376 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6q5qz_de9b3c28-175e-47ef-9138-4f455e701503/init-textfile/0.log" Jul 22 20:51:20.683107 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.683071 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q"] Jul 22 20:51:20.688591 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.688558 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.705578 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.705532 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q"] Jul 22 20:51:20.829134 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.829099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-sys\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.829319 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.829148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ztqj\" (UniqueName: \"kubernetes.io/projected/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-kube-api-access-4ztqj\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.829319 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.829255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-lib-modules\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.829435 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.829334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-proc\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.829435 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.829390 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-podres\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.930895 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.930843 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-sys\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.931102 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.930906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ztqj\" (UniqueName: \"kubernetes.io/projected/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-kube-api-access-4ztqj\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.931102 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.930959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-lib-modules\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.931102 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.931029 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-proc\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.931102 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.931082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-podres\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.931336 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.931245 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-podres\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.931336 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.931308 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-sys\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.931762 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.931731 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-lib-modules\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.931865 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.931803 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-proc\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:20.940029 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:20.939953 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ztqj\" (UniqueName: \"kubernetes.io/projected/5bcba8b4-12ba-44ab-a704-eb10a6d451d6-kube-api-access-4ztqj\") pod \"perf-node-gather-daemonset-qkm5q\" (UID: \"5bcba8b4-12ba-44ab-a704-eb10a6d451d6\") " pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:21.009357 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:21.009319 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:21.195954 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:21.195568 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q"] Jul 22 20:51:21.199806 ip-10-0-143-196 kubenswrapper[2575]: W0722 20:51:21.199773 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5bcba8b4_12ba_44ab_a704_eb10a6d451d6.slice/crio-9863b64e04d0827605bf7b5ba02515d39ccc240b2a00f015869ba65420717f86 WatchSource:0}: Error finding container 9863b64e04d0827605bf7b5ba02515d39ccc240b2a00f015869ba65420717f86: Status 404 returned error can't find the container with id 9863b64e04d0827605bf7b5ba02515d39ccc240b2a00f015869ba65420717f86 Jul 22 20:51:21.807205 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:21.807161 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" event={"ID":"5bcba8b4-12ba-44ab-a704-eb10a6d451d6","Type":"ContainerStarted","Data":"97e8f8279499f549308366f3031d687cb6d89a61e17a439b4ae870f453c73ad4"} Jul 22 20:51:21.807706 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:21.807231 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" event={"ID":"5bcba8b4-12ba-44ab-a704-eb10a6d451d6","Type":"ContainerStarted","Data":"9863b64e04d0827605bf7b5ba02515d39ccc240b2a00f015869ba65420717f86"} Jul 22 20:51:21.808030 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:21.808005 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:23.885810 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:23.885782 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xlzdp_a01377ab-7b6f-4775-a9ef-260b8216c7a4/dns/0.log" Jul 22 20:51:23.906199 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:23.906169 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xlzdp_a01377ab-7b6f-4775-a9ef-260b8216c7a4/kube-rbac-proxy/0.log" Jul 22 20:51:23.927628 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:23.927596 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f99nc_88aa1673-501e-43a1-8e35-20b2a70950d0/dns-node-resolver/0.log" Jul 22 20:51:24.535120 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:24.535080 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7b478545d6-q76ss_528a7576-71ce-438a-bfaa-e4e22be46b4d/registry/0.log" Jul 22 20:51:24.557511 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:24.557475 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5sdbv_a96e7282-4793-4de4-b2a0-deb12e53fb4c/node-ca/0.log" Jul 22 20:51:25.539570 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:25.539538 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46482l_cf0c6221-5afb-4a54-8ee5-c856d3a0e77d/istio-proxy/0.log" Jul 22 20:51:25.785301 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:25.785266 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5nc2r_cb12e370-8f88-49af-8643-ccffb2f996a6/discovery/0.log" Jul 22 20:51:26.791572 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:26.791539 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sw289_2f07cfd6-24f9-4d03-a5f6-5c4bc1c83116/serve-healthcheck-canary/0.log" Jul 22 20:51:27.346218 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:27.346179 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-btl22_677263c5-5fe3-40de-9004-02bb5bebf54b/kube-rbac-proxy/0.log" Jul 22 20:51:27.372317 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:27.372286 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-btl22_677263c5-5fe3-40de-9004-02bb5bebf54b/exporter/0.log" Jul 22 20:51:27.395764 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:27.395724 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-btl22_677263c5-5fe3-40de-9004-02bb5bebf54b/extractor/0.log" Jul 22 20:51:28.828049 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:28.828016 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" Jul 22 20:51:28.846026 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:28.845953 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nzwv7/perf-node-gather-daemonset-qkm5q" podStartSLOduration=8.845938441 podStartE2EDuration="8.845938441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 20:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 20:51:21.824905102 +0000 UTC m=+2982.337784809" watchObservedRunningTime="2026-07-22 20:51:28.845938441 +0000 UTC m=+2989.358818147" Jul 22 20:51:30.096754 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:30.096720 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64cc9776b5-sqx22_e2d423d9-17cc-4c4a-b492-3416c04b332c/manager/0.log" Jul 22 20:51:30.399233 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:30.399094 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9fc56f4f-78xpl_97c53fcf-3ae1-49af-a341-42aab4dfba86/manager/0.log" Jul 22 20:51:30.420577 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:30.420549 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9fc56f4f-78xpl_97c53fcf-3ae1-49af-a341-42aab4dfba86/copy-manifests/0.log" Jul 22 20:51:31.568036 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:31.567970 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46482l_cf0c6221-5afb-4a54-8ee5-c856d3a0e77d/istio-proxy/0.log" Jul 22 20:51:31.815725 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:31.815674 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5nc2r_cb12e370-8f88-49af-8643-ccffb2f996a6/discovery/0.log" Jul 22 20:51:31.923180 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:31.923141 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-76bfc99684-f8pnf_1bc578c9-18bc-4546-a051-a0ca2516502b/manager/0.log" Jul 22 20:51:33.164834 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:33.164805 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46482l_cf0c6221-5afb-4a54-8ee5-c856d3a0e77d/istio-proxy/0.log" Jul 22 20:51:33.390204 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:33.390115 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5nc2r_cb12e370-8f88-49af-8643-ccffb2f996a6/discovery/0.log" Jul 22 20:51:38.548699 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:38.548672 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jlwgk_205d0975-9993-4a3e-ae60-573dda5e4d73/kube-multus-additional-cni-plugins/0.log" Jul 22 20:51:38.568617 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:38.568580 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jlwgk_205d0975-9993-4a3e-ae60-573dda5e4d73/egress-router-binary-copy/0.log" Jul 22 20:51:38.587857 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:38.587825 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jlwgk_205d0975-9993-4a3e-ae60-573dda5e4d73/cni-plugins/0.log" Jul 22 20:51:38.606564 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:38.606536 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jlwgk_205d0975-9993-4a3e-ae60-573dda5e4d73/bond-cni-plugin/0.log" Jul 22 20:51:38.628770 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:38.628743 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jlwgk_205d0975-9993-4a3e-ae60-573dda5e4d73/routeoverride-cni/0.log" Jul 22 20:51:38.650881 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:38.650848 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jlwgk_205d0975-9993-4a3e-ae60-573dda5e4d73/whereabouts-cni-bincopy/0.log" Jul 22 20:51:38.671143 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:38.671117 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jlwgk_205d0975-9993-4a3e-ae60-573dda5e4d73/whereabouts-cni/0.log" Jul 22 20:51:38.921894 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:38.921811 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-b9gqf_21f357d8-3cda-47c1-b91b-1bc4fcf45113/kube-multus/0.log" Jul 22 20:51:38.997891 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:38.997852 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-29ngf_d77c3889-b5fa-491e-b122-3b1c06ac4bd0/network-metrics-daemon/0.log" Jul 22 20:51:39.021240 ip-10-0-143-196 kubenswrapper[2575]: I0722 20:51:39.021210 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-29ngf_d77c3889-b5fa-491e-b122-3b1c06ac4bd0/kube-rbac-proxy/0.log"