Jul 22 18:38:03.860989 ip-10-0-143-15 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 18:38:03.861005 ip-10-0-143-15 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 18:38:03.861015 ip-10-0-143-15 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 18:38:03.861364 ip-10-0-143-15 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 18:38:14.062731 ip-10-0-143-15 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 18:38:14.062746 ip-10-0-143-15 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 975377b7dea644fea8514109a31ae379 -- Jul 22 18:39:48.236608 ip-10-0-143-15 systemd[1]: Starting Kubernetes Kubelet... Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.580595 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585017 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585030 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585036 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585041 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585045 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585051 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585059 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 18:39:48.876973 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585064 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 18:39:48.646919 ip-10-0-143-15 systemd[1]: Started Kubernetes Kubelet. Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585068 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585071 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585075 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585079 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585083 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585087 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585091 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585095 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585098 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585102 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585106 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585110 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585114 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585117 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585121 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585125 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585129 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585133 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585138 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 18:39:48.890243 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585142 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585145 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585149 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585156 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585160 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585166 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585170 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585174 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585178 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585182 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585185 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585189 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585193 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585197 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585201 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585205 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585208 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585212 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585215 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585219 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 18:39:48.891084 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585223 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585226 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585230 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585233 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585237 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585241 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585244 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585247 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585251 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585255 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585258 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585263 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585267 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585271 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585275 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585281 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585300 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585305 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585309 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585314 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 18:39:48.891864 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585318 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585321 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585325 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585329 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585333 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585337 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585341 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585345 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585349 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585352 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585356 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585359 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585363 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585367 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585370 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585373 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585377 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585381 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585385 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 18:39:48.892501 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585388 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585392 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585397 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585402 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585407 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585411 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585414 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585419 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585423 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585426 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585430 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585434 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.585438 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587014 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587023 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587028 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587032 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587036 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587042 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587046 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 18:39:48.893110 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587051 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587055 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587059 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587063 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587067 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587071 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587076 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587080 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587084 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587088 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587092 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587096 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587102 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587108 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587111 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587115 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587119 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587122 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587126 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587131 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 18:39:48.893730 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587134 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587138 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587142 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587145 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587149 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587153 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587157 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587160 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587164 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587168 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587172 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587176 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587180 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587185 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587188 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587192 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587195 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587199 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587202 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587207 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 18:39:48.894571 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587211 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587214 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587218 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587222 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587227 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587232 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587236 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587240 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587244 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587248 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587252 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587255 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587259 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587263 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587266 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587270 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587274 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587279 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587298 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 18:39:48.895191 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587302 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587306 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587310 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587313 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587317 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587321 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587326 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587330 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587334 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587338 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587342 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587346 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587349 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587353 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587357 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587360 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587367 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587372 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587376 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 18:39:48.895825 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587380 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587384 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587388 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587392 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587396 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587399 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587404 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587408 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587413 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587417 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587421 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587425 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587429 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587432 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587938 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587945 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587950 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587954 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587958 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587962 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 18:39:48.896456 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587966 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587970 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587974 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587978 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587982 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587985 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587989 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587993 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.587996 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588000 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588004 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588008 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588012 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588015 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588019 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588023 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588027 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588031 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588035 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588038 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 18:39:48.897075 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588043 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588047 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588050 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588054 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588058 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588062 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588068 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588073 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588076 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588080 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588084 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588088 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588092 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588096 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588100 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588104 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588108 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588112 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588115 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588119 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 18:39:48.897711 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588122 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588126 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588130 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588134 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588137 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588141 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588145 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588148 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588152 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588155 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588159 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588165 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588170 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588174 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588179 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588183 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588187 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588191 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588196 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588200 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 18:39:48.898365 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588204 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588208 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588212 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588218 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588224 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588228 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588232 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588236 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588240 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588243 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588247 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588252 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588255 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588259 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588263 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588267 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588270 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588274 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588278 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 18:39:48.898916 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588282 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588302 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588306 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588310 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588314 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588318 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588324 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588331 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588335 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588339 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588343 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588348 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588352 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.588356 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588439 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588448 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588456 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588462 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588468 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588473 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 18:39:48.899552 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588479 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588484 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588488 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588493 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588497 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588502 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588507 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588511 2574 flags.go:64] FLAG: --cgroup-root="" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588516 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588520 2574 flags.go:64] FLAG: --client-ca-file="" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588524 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588528 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588534 2574 flags.go:64] FLAG: --cluster-domain="" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588542 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588546 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588551 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588555 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588561 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588565 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588569 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588574 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588578 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588582 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588588 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 18:39:48.900072 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588592 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588597 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588602 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588606 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588611 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588614 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588618 2574 flags.go:64] FLAG: --enable-server="true" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588623 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588630 2574 flags.go:64] FLAG: --event-burst="100" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588634 2574 flags.go:64] FLAG: --event-qps="50" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588638 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588643 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588647 2574 flags.go:64] FLAG: --eviction-hard="" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588652 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588657 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588661 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588665 2574 flags.go:64] FLAG: --eviction-soft="" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588670 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588674 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588678 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588682 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588688 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588692 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588696 2574 flags.go:64] FLAG: --feature-gates="" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588702 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 18:39:48.900939 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588706 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588710 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588715 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588720 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588724 2574 flags.go:64] FLAG: --help="false" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588728 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-143-15.ec2.internal" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588733 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588738 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588742 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588747 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588752 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588756 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588760 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588764 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588768 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588773 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588777 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588781 2574 flags.go:64] FLAG: --kube-reserved="" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588786 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588790 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588794 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588798 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588802 2574 flags.go:64] FLAG: --lock-file="" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588805 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 18:39:48.901655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588810 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588814 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588821 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588825 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588831 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588835 2574 flags.go:64] FLAG: --logging-format="text" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588839 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588844 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588848 2574 flags.go:64] FLAG: --manifest-url="" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588851 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588857 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588862 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588867 2574 flags.go:64] FLAG: --max-pods="110" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588872 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588877 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588882 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588886 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588890 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588894 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588899 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588908 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588912 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588916 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 18:39:48.902362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588921 2574 flags.go:64] FLAG: --pod-cidr="" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588926 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588933 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588937 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588941 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588945 2574 flags.go:64] FLAG: --port="10250" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588950 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588954 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02e1af2966618b30d" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588958 2574 flags.go:64] FLAG: --qos-reserved="" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588963 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588966 2574 flags.go:64] FLAG: --register-node="true" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588970 2574 flags.go:64] FLAG: --register-with-taints="" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588976 2574 flags.go:64] FLAG: --registry-burst="10" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588981 2574 flags.go:64] FLAG: --registry-qps="5" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588986 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588989 2574 flags.go:64] FLAG: --reserved-memory="" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588994 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.588999 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589003 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589007 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589011 2574 flags.go:64] FLAG: --runonce="false" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589015 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589020 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589024 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589030 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 18:39:48.903304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589034 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589038 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589043 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589047 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589051 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589056 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589059 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589063 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589068 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589073 2574 flags.go:64] FLAG: --system-cgroups="" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589077 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589084 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589088 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589092 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589098 2574 flags.go:64] FLAG: --tls-min-version="" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589102 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589106 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589110 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589114 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589119 2574 flags.go:64] FLAG: --v="2" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589126 2574 flags.go:64] FLAG: --version="false" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589131 2574 flags.go:64] FLAG: --vmodule="" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589136 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.589141 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589305 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 18:39:48.904076 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589310 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589315 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589319 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589323 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589327 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589330 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589336 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589339 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589343 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589347 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589351 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589354 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589358 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589362 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589366 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589369 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589373 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589378 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589382 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 18:39:48.977068 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589385 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589389 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589393 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589396 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589400 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589404 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589408 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589411 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589417 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589422 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589425 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589429 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589433 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589437 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589440 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589443 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589447 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589451 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589455 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589461 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 18:39:48.978127 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589465 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589468 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589472 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589475 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589479 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589483 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589486 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589490 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589494 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589498 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589501 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589506 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589510 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589513 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589517 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589520 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589524 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589527 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589531 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589535 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 18:39:48.979003 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589540 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589544 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589547 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589551 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589555 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589559 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589562 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589565 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589569 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589573 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589577 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589583 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589586 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589591 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589594 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589597 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589601 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589604 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589610 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589615 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 18:39:48.979910 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589619 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589624 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589628 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589632 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589638 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589644 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589649 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589653 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589657 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589661 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589666 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589669 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589675 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589678 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589682 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589687 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589692 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589698 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 18:39:48.980587 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.589703 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.590249 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.597659 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.597673 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.598249 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.598256 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598334 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598340 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598343 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598347 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598349 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598352 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598355 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598358 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598366 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 18:39:48.981107 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598369 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598372 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598374 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598382 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598385 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598388 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598391 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598393 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598396 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598399 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598401 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598404 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598406 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598409 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598411 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598414 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598417 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598420 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598422 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598425 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 18:39:48.981543 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598428 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598430 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598433 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598435 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598438 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598441 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598443 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598446 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598449 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598451 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598454 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598457 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598459 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598462 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598464 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598466 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598475 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598477 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598480 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598483 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 18:39:48.982211 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598485 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598488 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598491 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598493 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598496 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598498 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598502 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598506 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598511 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598514 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598517 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598520 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598524 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598527 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598529 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598533 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598535 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598538 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598541 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 18:39:48.982771 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598544 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598547 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598549 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598552 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598556 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598559 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598562 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598565 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598569 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598572 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598580 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598582 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598585 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598588 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598591 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598593 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598596 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598599 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598601 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 18:39:48.983386 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598604 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598606 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598612 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598614 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598617 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598619 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598622 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598624 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598628 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598632 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598635 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598637 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.598642 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598801 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598806 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 18:39:48.984079 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598809 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598812 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598814 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598817 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598820 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598823 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598828 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598832 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598835 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598843 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598846 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598849 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598852 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598854 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598857 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598860 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598863 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598866 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598868 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598871 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 18:39:48.984595 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598875 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598878 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598880 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598883 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598885 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598888 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598891 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598893 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598896 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598899 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598901 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598904 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598906 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598909 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598911 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598914 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598916 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598919 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598923 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 18:39:48.985122 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598926 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598928 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598931 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598939 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598942 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598944 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598947 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598949 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598952 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598956 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598959 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598962 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598965 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598969 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598972 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598976 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598979 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598982 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598985 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 18:39:48.985681 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598988 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598991 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598993 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598996 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.598999 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599001 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599004 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599007 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599009 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599012 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599014 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599017 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599020 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599023 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599026 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599028 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599031 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599039 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599042 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599044 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 18:39:48.986202 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599047 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599049 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599052 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599054 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599057 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599059 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599063 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599066 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599068 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599071 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599074 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599076 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599079 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599081 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599084 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599086 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599089 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599091 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 18:39:48.986896 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:48.599094 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 18:39:48.987576 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.599099 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 18:39:48.987576 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.599208 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 18:39:48.987576 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.601965 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 18:39:48.987576 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.603090 2574 server.go:1019] "Starting client certificate rotation" Jul 22 18:39:48.987576 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.603189 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 18:39:48.987576 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.603797 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 18:39:48.987576 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.606083 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 18:39:48.987576 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.607685 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 18:39:48.987576 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.615877 2574 log.go:25] "Validated CRI v1 runtime API" Jul 22 18:39:48.987576 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.617872 2574 log.go:25] "Validated CRI v1 image API" Jul 22 18:39:48.987576 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.619871 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 18:39:48.987576 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.621897 2574 fs.go:135] Filesystem UUIDs: map[44005ddf-94b7-4938-b191-6d66a0acaf99:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fcac9553-0b19-49b3-acf3-da39ea044f40:/dev/nvme0n1p4] Jul 22 18:39:48.988013 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.621910 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 18:39:48.988233 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.627033 2574 manager.go:216] Machine: {Timestamp:2026-07-22 18:39:48.625868451 +0000 UTC m=+0.274947875 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103258 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20bed54a9f994288b5d2ce96d52d94 SystemUUID:ec20bed5-4a9f-9942-88b5-d2ce96d52d94 BootID:975377b7-dea6-44fe-a851-4109a31ae379 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3e:92:f2:ff:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3e:92:f2:ff:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:3a:2a:0a:98:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 18:39:48.988233 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.627126 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 18:39:48.988233 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.627206 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 18:39:48.988233 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.629593 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 18:39:48.988233 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.629615 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-15.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 18:39:48.988233 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.630232 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 18:39:48.988233 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.630241 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 18:39:48.988233 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.630248 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.630904 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.632229 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.635570 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.635916 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.636043 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.636077 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.636094 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.636108 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.636118 2574 kubelet.go:398] "Adding apiserver pod source" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.636126 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.636546 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.636576 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.639727 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.641324 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.641353 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.641928 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.641941 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.641947 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.641953 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.641962 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.641968 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.642021 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.642027 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.642035 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.642041 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.642049 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.642057 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.642632 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.642639 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.646092 2574 server.go:1268] "Started kubelet" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.646181 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.646243 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.646790 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.648513 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.649129 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.649977 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-15.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.649969 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.650053 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 18:39:48.988664 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.652117 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.652929 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-15.ec2.internal.18c4b0a829b1b130 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-15.ec2.internal,UID:ip-10-0-143-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-15.ec2.internal,},FirstTimestamp:2026-07-22 18:39:48.646064432 +0000 UTC m=+0.295143856,LastTimestamp:2026-07-22 18:39:48.646064432 +0000 UTC m=+0.295143856,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-15.ec2.internal,}" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.655407 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.655417 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.655582 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.656058 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.656062 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.656083 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.656203 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.656214 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.656456 2574 factory.go:55] Registering systemd factory Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.656505 2574 factory.go:223] Registration of the systemd container factory successfully Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.656701 2574 factory.go:153] Registering CRI-O factory Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.656724 2574 factory.go:223] Registration of the crio container factory successfully Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.656788 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.656824 2574 factory.go:103] Registering Raw factory Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.656851 2574 manager.go:1196] Started watching for new ooms in manager Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.657515 2574 manager.go:349] Starting recovery of all containers Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.668426 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.668688 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.673837 2574 manager.go:354] Recovery completed Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.678063 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.683600 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.683633 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.683642 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.684088 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.684097 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.684100 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.684133 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.684852 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.684876 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:39:48.992673 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.684888 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.685934 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-15.ec2.internal.18c4b0a82beebde0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-15.ec2.internal,UID:ip-10-0-143-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-15.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-15.ec2.internal,},FirstTimestamp:2026-07-22 18:39:48.683619808 +0000 UTC m=+0.332699231,LastTimestamp:2026-07-22 18:39:48.683619808 +0000 UTC m=+0.332699231,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-15.ec2.internal,}" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.686238 2574 policy_none.go:49] "None policy: Start" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.686250 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.686259 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.687267 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6xd6w" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.688299 2574 policy_none.go:47] "Start" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.692987 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6xd6w" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.693975 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-15.ec2.internal.18c4b0a82bef040a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-15.ec2.internal,UID:ip-10-0-143-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-15.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-15.ec2.internal,},FirstTimestamp:2026-07-22 18:39:48.68363777 +0000 UTC m=+0.332717193,LastTimestamp:2026-07-22 18:39:48.68363777 +0000 UTC m=+0.332717193,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-15.ec2.internal,}" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.699779 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-15.ec2.internal.18c4b0a82bef2713 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-15.ec2.internal,UID:ip-10-0-143-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-15.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-15.ec2.internal,},FirstTimestamp:2026-07-22 18:39:48.683646739 +0000 UTC m=+0.332726164,LastTimestamp:2026-07-22 18:39:48.683646739 +0000 UTC m=+0.332726164,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-15.ec2.internal,}" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.736484 2574 manager.go:337] "Starting Device Plugin manager" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.736511 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.736521 2574 server.go:87] "Starting device plugin registration server" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.736747 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.736757 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.737312 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.737334 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.737377 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.737384 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.737393 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.796109 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 18:39:48.993959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.797229 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.797263 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.797294 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.797331 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.799312 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.837007 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.839901 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.839927 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.839939 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.839962 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.845711 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.845730 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-15.ec2.internal\": node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.868719 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.899563 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"] Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.899733 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.900515 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.900541 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.900558 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.902808 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.902906 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.902930 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.903892 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.903914 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.903926 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.903982 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.904003 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.904015 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.906197 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.906218 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.908869 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.908895 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:48.908907 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.967169 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-15.ec2.internal\" not found" node="ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.089822 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.977250 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:49.090814 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:48.981782 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-15.ec2.internal\" not found" node="ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.090814 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.077312 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5982cdd9aa07454c3773d04a38a8a229-config\") pod \"kube-apiserver-proxy-ip-10-0-143-15.ec2.internal\" (UID: \"5982cdd9aa07454c3773d04a38a8a229\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.090814 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.077345 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/749a592c7f4bbbb81e4b1740a322d6c0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"749a592c7f4bbbb81e4b1740a322d6c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.090814 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.077366 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/749a592c7f4bbbb81e4b1740a322d6c0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"749a592c7f4bbbb81e4b1740a322d6c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.090814 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:49.077413 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:49.179323 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:49.178023 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:49.179323 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.178094 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5982cdd9aa07454c3773d04a38a8a229-config\") pod \"kube-apiserver-proxy-ip-10-0-143-15.ec2.internal\" (UID: \"5982cdd9aa07454c3773d04a38a8a229\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.179323 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.178126 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/749a592c7f4bbbb81e4b1740a322d6c0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"749a592c7f4bbbb81e4b1740a322d6c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.179323 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.178151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/749a592c7f4bbbb81e4b1740a322d6c0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"749a592c7f4bbbb81e4b1740a322d6c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.179323 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.178198 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5982cdd9aa07454c3773d04a38a8a229-config\") pod \"kube-apiserver-proxy-ip-10-0-143-15.ec2.internal\" (UID: \"5982cdd9aa07454c3773d04a38a8a229\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.179323 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.178210 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/749a592c7f4bbbb81e4b1740a322d6c0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"749a592c7f4bbbb81e4b1740a322d6c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.179323 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.178214 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/749a592c7f4bbbb81e4b1740a322d6c0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"749a592c7f4bbbb81e4b1740a322d6c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.312609 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.269396 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.312609 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:49.278999 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:49.312609 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.285157 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" Jul 22 18:39:49.401250 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:49.380114 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:49.548887 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:49.480463 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:49.548887 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.509884 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 18:39:49.637324 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:49.580808 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:49.637324 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.602956 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 18:39:49.637324 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.603120 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 18:39:49.637324 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.603148 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 18:39:49.720207 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:49.680954 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:49.720207 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.699490 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 18:34:49 +0000 UTC" deadline="2028-04-29 22:03:21.604590664 +0000 UTC" Jul 22 18:39:49.720207 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.699511 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15531h23m31.905083116s" Jul 22 18:39:49.790954 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.743886 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 18:39:49.790954 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:49.781661 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:49.857612 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.855863 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:39:51.135100 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.866749 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:39:51.135100 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.866782 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:39:51.135100 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:49.866797 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:39:51.135100 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:49.881960 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:51.212299 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:51.199258 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found" Jul 22 18:39:51.302417 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.215790 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 18:39:51.302417 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.261085 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" Jul 22 18:39:51.302417 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.273041 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 18:39:51.302417 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.273811 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" Jul 22 18:39:51.302417 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.281716 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 18:39:51.395911 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:51.310389 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749a592c7f4bbbb81e4b1740a322d6c0.slice/crio-445cf9e76ae32cfa8bdd5b4be2a8d9ef9f4f2d3779fa41ed26c4c663757da01d WatchSource:0}: Error finding container 445cf9e76ae32cfa8bdd5b4be2a8d9ef9f4f2d3779fa41ed26c4c663757da01d: Status 404 returned error can't find the container with id 445cf9e76ae32cfa8bdd5b4be2a8d9ef9f4f2d3779fa41ed26c4c663757da01d Jul 22 18:39:51.395911 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.314098 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 18:39:51.839184 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.839106 2574 apiserver.go:52] "Watching apiserver" Jul 22 18:39:51.845917 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.845893 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 18:39:51.846394 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.846370 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-8n5kx","kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q","openshift-cluster-node-tuning-operator/tuned-vtc9x","openshift-image-registry/node-ca-gg7vx","openshift-multus/multus-zt4fh","openshift-multus/network-metrics-daemon-lc7qt","openshift-network-diagnostics/network-check-target-cvmqq","kube-system/global-pull-secret-syncer-9xjws","openshift-dns/node-resolver-j7kgx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal","openshift-multus/multus-additional-cni-plugins-8ztgb","openshift-network-operator/iptables-alerter-fw9m4","openshift-ovn-kubernetes/ovnkube-node-7s8l6"] Jul 22 18:39:51.848870 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.848848 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:39:51.848954 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:51.848921 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:39:51.851517 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.851490 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fw9m4" Jul 22 18:39:51.853872 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.853854 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 18:39:51.854130 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.854114 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 18:39:51.854332 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.854317 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 18:39:51.854487 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.854473 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5dxfn\"" Jul 22 18:39:51.854674 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.854662 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:51.856043 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.856019 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.856672 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.856646 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 18:39:51.858393 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.858245 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 18:39:51.859049 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.858638 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gg7vx" Jul 22 18:39:51.859049 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.858846 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6pdjq\"" Jul 22 18:39:51.859180 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.859071 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 18:39:51.859180 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.859133 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cvqqv\"" Jul 22 18:39:51.859288 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.859236 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 18:39:51.861251 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.861169 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zt4fh" Jul 22 18:39:51.861315 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.861298 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 18:39:51.861658 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.861611 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 18:39:51.861658 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.861647 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 18:39:51.861804 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.861750 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 18:39:51.862011 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.861993 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 18:39:51.862520 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.862504 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qjfp\"" Jul 22 18:39:51.863488 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.863469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:51.863574 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:51.863540 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:39:51.863574 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.863567 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 18:39:51.864089 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.863833 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 18:39:51.864089 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.863929 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 18:39:51.864089 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.863957 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rcmcf\"" Jul 22 18:39:51.864301 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.864199 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 18:39:51.865706 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.865689 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8n5kx" Jul 22 18:39:51.867491 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.867461 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 18:39:51.867776 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.867760 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 18:39:51.867776 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.867771 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sk2ts\"" Jul 22 18:39:51.868446 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.868167 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:51.870525 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.870510 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7kgx" Jul 22 18:39:51.870875 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.870838 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 18:39:51.871181 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.871162 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qjgq6\"" Jul 22 18:39:51.871270 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.871219 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 18:39:51.871370 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.871355 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 18:39:51.871423 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.871395 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 18:39:51.871474 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.871463 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 18:39:51.871581 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.871567 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 18:39:51.872755 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.872737 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 18:39:51.872846 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.872815 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:39:51.872906 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:51.872867 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:39:51.873297 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.873173 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4sqbs\"" Jul 22 18:39:51.873297 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.873221 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 18:39:51.874328 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.874309 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 18:39:51.875368 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.875351 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:51.877352 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.877335 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q86fk\"" Jul 22 18:39:51.877671 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.877652 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 18:39:51.877758 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.877749 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 18:39:51.891856 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.891827 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-24jpl" Jul 22 18:39:51.900917 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.900900 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-24jpl" Jul 22 18:39:51.910638 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.910050 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-var-lib-kubelet\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.910638 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.910481 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9mtn\" (UniqueName: \"kubernetes.io/projected/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-kube-api-access-h9mtn\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.910638 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.910522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f12f5040-f33c-4fbe-87a5-f95391205cd5-iptables-alerter-script\") pod \"iptables-alerter-fw9m4\" (UID: \"f12f5040-f33c-4fbe-87a5-f95391205cd5\") " pod="openshift-network-operator/iptables-alerter-fw9m4" Jul 22 18:39:51.910638 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.910559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f12f5040-f33c-4fbe-87a5-f95391205cd5-host-slash\") pod \"iptables-alerter-fw9m4\" (UID: \"f12f5040-f33c-4fbe-87a5-f95391205cd5\") " pod="openshift-network-operator/iptables-alerter-fw9m4" Jul 22 18:39:51.910638 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.910590 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-run\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.910638 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.910629 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-tmp\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.911130 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.910663 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8t9d\" (UniqueName: \"kubernetes.io/projected/4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4-kube-api-access-c8t9d\") pod \"node-ca-gg7vx\" (UID: \"4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4\") " pod="openshift-image-registry/node-ca-gg7vx" Jul 22 18:39:51.911130 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.910692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4-serviceca\") pod \"node-ca-gg7vx\" (UID: \"4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4\") " pod="openshift-image-registry/node-ca-gg7vx" Jul 22 18:39:51.911130 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.910723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:51.911130 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.910884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-registration-dir\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:51.911130 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.910916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-tuned\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.911130 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.910945 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-kubernetes\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.911432 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911195 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:39:51.911432 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911239 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-device-dir\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:51.911432 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911295 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-sys-fs\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:51.911432 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-host\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.911432 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911376 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4-host\") pod \"node-ca-gg7vx\" (UID: \"4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4\") " pod="openshift-image-registry/node-ca-gg7vx" Jul 22 18:39:51.911432 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911417 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-socket-dir\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:51.911764 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911462 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-sysconfig\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.911764 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-sysctl-d\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.911764 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911528 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-sysctl-conf\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.911764 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911580 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-systemd\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.911764 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911615 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c62mh\" (UniqueName: \"kubernetes.io/projected/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-kube-api-access-c62mh\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:39:51.911764 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911652 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h6kb\" (UniqueName: \"kubernetes.io/projected/f12f5040-f33c-4fbe-87a5-f95391205cd5-kube-api-access-6h6kb\") pod \"iptables-alerter-fw9m4\" (UID: \"f12f5040-f33c-4fbe-87a5-f95391205cd5\") " pod="openshift-network-operator/iptables-alerter-fw9m4" Jul 22 18:39:51.911764 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-modprobe-d\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.911764 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911713 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-sys\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.912113 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-lib-modules\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:51.912113 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911828 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-etc-selinux\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:51.912113 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.911876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnwmh\" (UniqueName: \"kubernetes.io/projected/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-kube-api-access-xnwmh\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:51.958543 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:51.958517 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 18:39:52.006239 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.006181 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" event={"ID":"749a592c7f4bbbb81e4b1740a322d6c0","Type":"ContainerStarted","Data":"445cf9e76ae32cfa8bdd5b4be2a8d9ef9f4f2d3779fa41ed26c4c663757da01d"} Jul 22 18:39:52.012488 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012458 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-var-lib-kubelet\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.012630 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012498 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f37a932-854e-4799-894e-db1e2b8a27d3-dbus\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:52.012630 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-systemd-units\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.012630 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-slash\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.012630 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-cni-bin\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.012630 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012626 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.012840 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012646 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73a193b7-d3c9-451c-af5c-fe08756dba3f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.012840 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012679 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.012840 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-registration-dir\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.012840 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-tuned\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.012840 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012738 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f37a932-854e-4799-894e-db1e2b8a27d3-kubelet-config\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:52.012840 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012770 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-var-lib-openvswitch\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.012840 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012785 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-registration-dir\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.012840 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-os-release\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.012840 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012798 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.012840 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-run-multus-certs\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012852 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/73a193b7-d3c9-451c-af5c-fe08756dba3f-os-release\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012913 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-kubernetes\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.012948 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-run-netns\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013000 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-kubernetes\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-var-lib-cni-multus\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73a193b7-d3c9-451c-af5c-fe08756dba3f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013139 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013143 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-run-ovn-kubernetes\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-socket-dir\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013244 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-sysctl-d\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-systemd\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013274 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-run-netns\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.013294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013296 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-run-openvswitch\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e953389b-d0c5-4bc7-907a-1687430ff7ba-multus-daemon-config\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c62mh\" (UniqueName: \"kubernetes.io/projected/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-kube-api-access-c62mh\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013359 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-systemd\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h6kb\" (UniqueName: \"kubernetes.io/projected/f12f5040-f33c-4fbe-87a5-f95391205cd5-kube-api-access-6h6kb\") pod \"iptables-alerter-fw9m4\" (UID: \"f12f5040-f33c-4fbe-87a5-f95391205cd5\") " pod="openshift-network-operator/iptables-alerter-fw9m4" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013402 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-modprobe-d\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013404 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-sysctl-d\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013432 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64894f60-a8ee-4da7-a605-e109bb3a84aa-agent-certs\") pod \"konnectivity-agent-8n5kx\" (UID: \"64894f60-a8ee-4da7-a605-e109bb3a84aa\") " pod="kube-system/konnectivity-agent-8n5kx" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013471 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-cni-netd\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013495 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hwg9\" (UniqueName: \"kubernetes.io/projected/daf45312-be61-4ecc-bf68-ee249f228c2b-kube-api-access-6hwg9\") pod \"node-resolver-j7kgx\" (UID: \"daf45312-be61-4ecc-bf68-ee249f228c2b\") " pod="openshift-dns/node-resolver-j7kgx" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013518 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-multus-conf-dir\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013534 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-socket-dir\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013541 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zznkf\" (UniqueName: \"kubernetes.io/projected/e953389b-d0c5-4bc7-907a-1687430ff7ba-kube-api-access-zznkf\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013566 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-modprobe-d\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.013960 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013573 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-etc-selinux\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013632 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-var-lib-kubelet\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013645 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-etc-selinux\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9mtn\" (UniqueName: \"kubernetes.io/projected/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-kube-api-access-h9mtn\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013701 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-var-lib-kubelet\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013778 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-kubelet\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013808 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bfpp\" (UniqueName: \"kubernetes.io/projected/a87719eb-a436-4abc-99f3-db0c8db21201-kube-api-access-2bfpp\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013849 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f12f5040-f33c-4fbe-87a5-f95391205cd5-host-slash\") pod \"iptables-alerter-fw9m4\" (UID: \"f12f5040-f33c-4fbe-87a5-f95391205cd5\") " pod="openshift-network-operator/iptables-alerter-fw9m4" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-tmp\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013885 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f12f5040-f33c-4fbe-87a5-f95391205cd5-host-slash\") pod \"iptables-alerter-fw9m4\" (UID: \"f12f5040-f33c-4fbe-87a5-f95391205cd5\") " pod="openshift-network-operator/iptables-alerter-fw9m4" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013896 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-etc-openvswitch\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013936 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a87719eb-a436-4abc-99f3-db0c8db21201-ovn-node-metrics-cert\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a87719eb-a436-4abc-99f3-db0c8db21201-ovnkube-script-lib\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.013999 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgl5v\" (UniqueName: \"kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v\") pod \"network-check-target-cvmqq\" (UID: \"cee0ab35-2e33-4a5d-83fe-c08b51c0ff47\") " pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:39:52.014531 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4-serviceca\") pod \"node-ca-gg7vx\" (UID: \"4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4\") " pod="openshift-image-registry/node-ca-gg7vx" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64894f60-a8ee-4da7-a605-e109bb3a84aa-konnectivity-ca\") pod \"konnectivity-agent-8n5kx\" (UID: \"64894f60-a8ee-4da7-a605-e109bb3a84aa\") " pod="kube-system/konnectivity-agent-8n5kx" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014087 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73a193b7-d3c9-451c-af5c-fe08756dba3f-cnibin\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-etc-kubernetes\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/73a193b7-d3c9-451c-af5c-fe08756dba3f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014159 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-system-cni-dir\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e953389b-d0c5-4bc7-907a-1687430ff7ba-cni-binary-copy\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014205 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-multus-socket-dir-parent\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-run-k8s-cni-cncf-io\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-var-lib-cni-bin\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a87719eb-a436-4abc-99f3-db0c8db21201-ovnkube-config\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014352 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014378 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-device-dir\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-sys-fs\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.015235 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-host\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.014450 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014485 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4-host\") pod \"node-ca-gg7vx\" (UID: \"4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4\") " pod="openshift-image-registry/node-ca-gg7vx" Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4-host\") pod \"node-ca-gg7vx\" (UID: \"4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4\") " pod="openshift-image-registry/node-ca-gg7vx" Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014509 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-device-dir\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.014535 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs podName:d27e81e3-a9d6-4c4b-b138-3a4394204eb3 nodeName:}" failed. No retries permitted until 2026-07-22 18:39:52.514502375 +0000 UTC m=+2.961897590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs") pod "network-metrics-daemon-lc7qt" (UID: "d27e81e3-a9d6-4c4b-b138-3a4394204eb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014566 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-node-log\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014576 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-host\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014594 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-log-socket\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014624 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-sys-fs\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-sysconfig\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014662 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-sysctl-conf\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014677 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-sysconfig\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014690 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-run-systemd\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.015907 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014714 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/daf45312-be61-4ecc-bf68-ee249f228c2b-hosts-file\") pod \"node-resolver-j7kgx\" (UID: \"daf45312-be61-4ecc-bf68-ee249f228c2b\") " pod="openshift-dns/node-resolver-j7kgx" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014739 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-multus-cni-dir\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73a193b7-d3c9-451c-af5c-fe08756dba3f-system-cni-dir\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014786 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a87719eb-a436-4abc-99f3-db0c8db21201-env-overrides\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014787 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-sysctl-conf\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-sys\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014834 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-lib-modules\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-run-ovn\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014878 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-cnibin\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-sys\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnwmh\" (UniqueName: \"kubernetes.io/projected/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-kube-api-access-xnwmh\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-hostroot\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014944 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-lib-modules\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/73a193b7-d3c9-451c-af5c-fe08756dba3f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.016389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.014996 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mffn\" (UniqueName: \"kubernetes.io/projected/73a193b7-d3c9-451c-af5c-fe08756dba3f-kube-api-access-9mffn\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.017106 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.015024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f12f5040-f33c-4fbe-87a5-f95391205cd5-iptables-alerter-script\") pod \"iptables-alerter-fw9m4\" (UID: \"f12f5040-f33c-4fbe-87a5-f95391205cd5\") " pod="openshift-network-operator/iptables-alerter-fw9m4" Jul 22 18:39:52.017106 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.015046 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-run\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.017106 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.015085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8t9d\" (UniqueName: \"kubernetes.io/projected/4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4-kube-api-access-c8t9d\") pod \"node-ca-gg7vx\" (UID: \"4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4\") " pod="openshift-image-registry/node-ca-gg7vx" Jul 22 18:39:52.017106 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.015109 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf45312-be61-4ecc-bf68-ee249f228c2b-tmp-dir\") pod \"node-resolver-j7kgx\" (UID: \"daf45312-be61-4ecc-bf68-ee249f228c2b\") " pod="openshift-dns/node-resolver-j7kgx" Jul 22 18:39:52.017106 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.015220 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-run\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.017106 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.015582 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f12f5040-f33c-4fbe-87a5-f95391205cd5-iptables-alerter-script\") pod \"iptables-alerter-fw9m4\" (UID: \"f12f5040-f33c-4fbe-87a5-f95391205cd5\") " pod="openshift-network-operator/iptables-alerter-fw9m4" Jul 22 18:39:52.017106 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.015715 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4-serviceca\") pod \"node-ca-gg7vx\" (UID: \"4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4\") " pod="openshift-image-registry/node-ca-gg7vx" Jul 22 18:39:52.017106 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.016894 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-etc-tuned\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.017106 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.016926 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-tmp\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.028361 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.028323 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 18:39:52.028890 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.028868 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8t9d\" (UniqueName: \"kubernetes.io/projected/4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4-kube-api-access-c8t9d\") pod \"node-ca-gg7vx\" (UID: \"4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4\") " pod="openshift-image-registry/node-ca-gg7vx" Jul 22 18:39:52.028993 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.028931 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnwmh\" (UniqueName: \"kubernetes.io/projected/e9e9199d-7044-4862-a13f-0cc2ef4ee32a-kube-api-access-xnwmh\") pod \"aws-ebs-csi-driver-node-f9s2q\" (UID: \"e9e9199d-7044-4862-a13f-0cc2ef4ee32a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.028993 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.028958 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9mtn\" (UniqueName: \"kubernetes.io/projected/96cfe8d7-79cd-41ca-8f7b-10e116d5a85d-kube-api-access-h9mtn\") pod \"tuned-vtc9x\" (UID: \"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d\") " pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.029108 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.028997 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h6kb\" (UniqueName: \"kubernetes.io/projected/f12f5040-f33c-4fbe-87a5-f95391205cd5-kube-api-access-6h6kb\") pod \"iptables-alerter-fw9m4\" (UID: \"f12f5040-f33c-4fbe-87a5-f95391205cd5\") " pod="openshift-network-operator/iptables-alerter-fw9m4" Jul 22 18:39:52.029406 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.029386 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c62mh\" (UniqueName: \"kubernetes.io/projected/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-kube-api-access-c62mh\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:39:52.115916 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.115821 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-run-netns\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.115916 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.115878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-var-lib-cni-multus\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.115916 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.115896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73a193b7-d3c9-451c-af5c-fe08756dba3f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.115916 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.115915 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:52.116177 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.115928 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-run-netns\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.116177 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.115937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-run-ovn-kubernetes\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.116177 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.115949 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-var-lib-cni-multus\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.116177 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.115963 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-run-ovn-kubernetes\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.116177 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.115979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-run-netns\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.116427 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116017 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-run-openvswitch\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.116427 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116122 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-run-openvswitch\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.116427 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116378 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e953389b-d0c5-4bc7-907a-1687430ff7ba-multus-daemon-config\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.116427 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.116397 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:39:52.116596 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116454 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64894f60-a8ee-4da7-a605-e109bb3a84aa-agent-certs\") pod \"konnectivity-agent-8n5kx\" (UID: \"64894f60-a8ee-4da7-a605-e109bb3a84aa\") " pod="kube-system/konnectivity-agent-8n5kx" Jul 22 18:39:52.116596 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.116473 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret podName:0f37a932-854e-4799-894e-db1e2b8a27d3 nodeName:}" failed. No retries permitted until 2026-07-22 18:39:52.616453559 +0000 UTC m=+3.063848771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret") pod "global-pull-secret-syncer-9xjws" (UID: "0f37a932-854e-4799-894e-db1e2b8a27d3") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:39:52.116596 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116496 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-run-netns\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.116596 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-cni-netd\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.116596 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116545 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73a193b7-d3c9-451c-af5c-fe08756dba3f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.116596 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116553 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-cni-netd\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.116596 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116546 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hwg9\" (UniqueName: \"kubernetes.io/projected/daf45312-be61-4ecc-bf68-ee249f228c2b-kube-api-access-6hwg9\") pod \"node-resolver-j7kgx\" (UID: \"daf45312-be61-4ecc-bf68-ee249f228c2b\") " pod="openshift-dns/node-resolver-j7kgx" Jul 22 18:39:52.116915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-multus-conf-dir\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.116915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116646 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-multus-conf-dir\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.116915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116679 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zznkf\" (UniqueName: \"kubernetes.io/projected/e953389b-d0c5-4bc7-907a-1687430ff7ba-kube-api-access-zznkf\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.116915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116711 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-kubelet\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.116915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfpp\" (UniqueName: \"kubernetes.io/projected/a87719eb-a436-4abc-99f3-db0c8db21201-kube-api-access-2bfpp\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.116915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116775 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-etc-openvswitch\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.116915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a87719eb-a436-4abc-99f3-db0c8db21201-ovn-node-metrics-cert\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.116915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116834 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a87719eb-a436-4abc-99f3-db0c8db21201-ovnkube-script-lib\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.116915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgl5v\" (UniqueName: \"kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v\") pod \"network-check-target-cvmqq\" (UID: \"cee0ab35-2e33-4a5d-83fe-c08b51c0ff47\") " pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:39:52.116915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.116887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-kubelet\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.117390 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.117091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-etc-openvswitch\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.117390 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.117271 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e953389b-d0c5-4bc7-907a-1687430ff7ba-multus-daemon-config\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.117901 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.117741 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64894f60-a8ee-4da7-a605-e109bb3a84aa-konnectivity-ca\") pod \"konnectivity-agent-8n5kx\" (UID: \"64894f60-a8ee-4da7-a605-e109bb3a84aa\") " pod="kube-system/konnectivity-agent-8n5kx" Jul 22 18:39:52.117901 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.117792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73a193b7-d3c9-451c-af5c-fe08756dba3f-cnibin\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.117901 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.117826 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-etc-kubernetes\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.117901 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.117861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/73a193b7-d3c9-451c-af5c-fe08756dba3f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.117901 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.117895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-system-cni-dir\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.118200 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.117926 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e953389b-d0c5-4bc7-907a-1687430ff7ba-cni-binary-copy\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.118200 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.117952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-multus-socket-dir-parent\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.118200 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.117998 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-run-k8s-cni-cncf-io\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.118200 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-var-lib-cni-bin\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.118200 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118065 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a87719eb-a436-4abc-99f3-db0c8db21201-ovnkube-config\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.118200 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-node-log\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.118200 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118143 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-log-socket\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.118200 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118147 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a87719eb-a436-4abc-99f3-db0c8db21201-ovnkube-script-lib\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.118573 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-run-systemd\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.118573 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/daf45312-be61-4ecc-bf68-ee249f228c2b-hosts-file\") pod \"node-resolver-j7kgx\" (UID: \"daf45312-be61-4ecc-bf68-ee249f228c2b\") " pod="openshift-dns/node-resolver-j7kgx" Jul 22 18:39:52.118573 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-multus-cni-dir\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.118573 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73a193b7-d3c9-451c-af5c-fe08756dba3f-system-cni-dir\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.118573 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a87719eb-a436-4abc-99f3-db0c8db21201-env-overrides\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.118573 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-run-ovn\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.118573 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-cnibin\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.118573 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-hostroot\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.118573 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/73a193b7-d3c9-451c-af5c-fe08756dba3f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.118573 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118524 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mffn\" (UniqueName: \"kubernetes.io/projected/73a193b7-d3c9-451c-af5c-fe08756dba3f-kube-api-access-9mffn\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.118573 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf45312-be61-4ecc-bf68-ee249f228c2b-tmp-dir\") pod \"node-resolver-j7kgx\" (UID: \"daf45312-be61-4ecc-bf68-ee249f228c2b\") " pod="openshift-dns/node-resolver-j7kgx" Jul 22 18:39:52.118573 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118573 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e953389b-d0c5-4bc7-907a-1687430ff7ba-cni-binary-copy\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.119094 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118588 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-var-lib-kubelet\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.119094 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.118994 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-run-systemd\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.119094 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119060 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-node-log\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.119226 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119116 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-log-socket\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.119226 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119131 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a87719eb-a436-4abc-99f3-db0c8db21201-ovnkube-config\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.119226 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119171 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64894f60-a8ee-4da7-a605-e109bb3a84aa-konnectivity-ca\") pod \"konnectivity-agent-8n5kx\" (UID: \"64894f60-a8ee-4da7-a605-e109bb3a84aa\") " pod="kube-system/konnectivity-agent-8n5kx" Jul 22 18:39:52.119226 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/daf45312-be61-4ecc-bf68-ee249f228c2b-hosts-file\") pod \"node-resolver-j7kgx\" (UID: \"daf45312-be61-4ecc-bf68-ee249f228c2b\") " pod="openshift-dns/node-resolver-j7kgx" Jul 22 18:39:52.119394 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119348 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-multus-cni-dir\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.119445 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119396 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73a193b7-d3c9-451c-af5c-fe08756dba3f-system-cni-dir\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.119494 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119457 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-cnibin\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.119542 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119504 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-run-ovn\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.119587 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119550 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a87719eb-a436-4abc-99f3-db0c8db21201-env-overrides\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119626 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-var-lib-cni-bin\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119641 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/73a193b7-d3c9-451c-af5c-fe08756dba3f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119683 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73a193b7-d3c9-451c-af5c-fe08756dba3f-cnibin\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119709 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-hostroot\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119755 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-var-lib-kubelet\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119759 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf45312-be61-4ecc-bf68-ee249f228c2b-tmp-dir\") pod \"node-resolver-j7kgx\" (UID: \"daf45312-be61-4ecc-bf68-ee249f228c2b\") " pod="openshift-dns/node-resolver-j7kgx" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119810 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-multus-socket-dir-parent\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-etc-kubernetes\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f37a932-854e-4799-894e-db1e2b8a27d3-dbus\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-systemd-units\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119922 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-slash\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.119954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-cni-bin\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73a193b7-d3c9-451c-af5c-fe08756dba3f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.120461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120072 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f37a932-854e-4799-894e-db1e2b8a27d3-kubelet-config\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-var-lib-openvswitch\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120111 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-run-k8s-cni-cncf-io\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120137 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-os-release\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120154 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-cni-bin\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-run-multus-certs\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120180 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/73a193b7-d3c9-451c-af5c-fe08756dba3f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/73a193b7-d3c9-451c-af5c-fe08756dba3f-os-release\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120222 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f37a932-854e-4799-894e-db1e2b8a27d3-dbus\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120267 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-system-cni-dir\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120277 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-systemd-units\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120313 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/73a193b7-d3c9-451c-af5c-fe08756dba3f-os-release\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-slash\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120338 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f37a932-854e-4799-894e-db1e2b8a27d3-kubelet-config\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120377 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-host-run-multus-certs\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.121242 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120425 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-var-lib-openvswitch\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.121917 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120470 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e953389b-d0c5-4bc7-907a-1687430ff7ba-os-release\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.121917 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120618 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87719eb-a436-4abc-99f3-db0c8db21201-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.121917 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.120888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73a193b7-d3c9-451c-af5c-fe08756dba3f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.122279 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.122186 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64894f60-a8ee-4da7-a605-e109bb3a84aa-agent-certs\") pod \"konnectivity-agent-8n5kx\" (UID: \"64894f60-a8ee-4da7-a605-e109bb3a84aa\") " pod="kube-system/konnectivity-agent-8n5kx" Jul 22 18:39:52.123034 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.123016 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a87719eb-a436-4abc-99f3-db0c8db21201-ovn-node-metrics-cert\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.127794 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.127777 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:39:52.127931 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.127798 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:39:52.127931 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.127827 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mgl5v for pod openshift-network-diagnostics/network-check-target-cvmqq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:39:52.127931 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.127899 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v podName:cee0ab35-2e33-4a5d-83fe-c08b51c0ff47 nodeName:}" failed. No retries permitted until 2026-07-22 18:39:52.627875072 +0000 UTC m=+3.075270306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mgl5v" (UniqueName: "kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v") pod "network-check-target-cvmqq" (UID: "cee0ab35-2e33-4a5d-83fe-c08b51c0ff47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:39:52.130536 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.130515 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bfpp\" (UniqueName: \"kubernetes.io/projected/a87719eb-a436-4abc-99f3-db0c8db21201-kube-api-access-2bfpp\") pod \"ovnkube-node-7s8l6\" (UID: \"a87719eb-a436-4abc-99f3-db0c8db21201\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.130646 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.130585 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hwg9\" (UniqueName: \"kubernetes.io/projected/daf45312-be61-4ecc-bf68-ee249f228c2b-kube-api-access-6hwg9\") pod \"node-resolver-j7kgx\" (UID: \"daf45312-be61-4ecc-bf68-ee249f228c2b\") " pod="openshift-dns/node-resolver-j7kgx" Jul 22 18:39:52.131043 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.131019 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mffn\" (UniqueName: \"kubernetes.io/projected/73a193b7-d3c9-451c-af5c-fe08756dba3f-kube-api-access-9mffn\") pod \"multus-additional-cni-plugins-8ztgb\" (UID: \"73a193b7-d3c9-451c-af5c-fe08756dba3f\") " pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.131338 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.131318 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zznkf\" (UniqueName: \"kubernetes.io/projected/e953389b-d0c5-4bc7-907a-1687430ff7ba-kube-api-access-zznkf\") pod \"multus-zt4fh\" (UID: \"e953389b-d0c5-4bc7-907a-1687430ff7ba\") " pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.165863 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.165841 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fw9m4" Jul 22 18:39:52.174036 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.174009 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" Jul 22 18:39:52.181667 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.181644 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" Jul 22 18:39:52.189247 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.189230 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gg7vx" Jul 22 18:39:52.194704 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.194688 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zt4fh" Jul 22 18:39:52.202278 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.202251 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8n5kx" Jul 22 18:39:52.208927 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.208911 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:39:52.217528 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.217509 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7kgx" Jul 22 18:39:52.222142 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.222123 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8ztgb" Jul 22 18:39:52.485413 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:52.485379 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3a6ef6_5c0f_4077_9228_9fbd12b1c5f4.slice/crio-af9e9d72c165e1de71e21a6a3ab23ca924d4bb5555ea9f0d852b13f0d09b4f71 WatchSource:0}: Error finding container af9e9d72c165e1de71e21a6a3ab23ca924d4bb5555ea9f0d852b13f0d09b4f71: Status 404 returned error can't find the container with id af9e9d72c165e1de71e21a6a3ab23ca924d4bb5555ea9f0d852b13f0d09b4f71 Jul 22 18:39:52.486302 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:52.486272 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cfe8d7_79cd_41ca_8f7b_10e116d5a85d.slice/crio-01a3f50b7a008f95c0a6207faf99f3b039a71bc5189f628bc4d8a887f3a292ae WatchSource:0}: Error finding container 01a3f50b7a008f95c0a6207faf99f3b039a71bc5189f628bc4d8a887f3a292ae: Status 404 returned error can't find the container with id 01a3f50b7a008f95c0a6207faf99f3b039a71bc5189f628bc4d8a887f3a292ae Jul 22 18:39:52.490153 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:52.490129 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64894f60_a8ee_4da7_a605_e109bb3a84aa.slice/crio-c2ac8de155701de97dd913ee50b8501059a9eb0dff645e49232854f3aa9f20ea WatchSource:0}: Error finding container c2ac8de155701de97dd913ee50b8501059a9eb0dff645e49232854f3aa9f20ea: Status 404 returned error can't find the container with id c2ac8de155701de97dd913ee50b8501059a9eb0dff645e49232854f3aa9f20ea Jul 22 18:39:52.490893 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:52.490816 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12f5040_f33c_4fbe_87a5_f95391205cd5.slice/crio-fd780bf2654208cd1c96ea7c0421d20fd35f3946d63f4a8a00a669b310d3d8a2 WatchSource:0}: Error finding container fd780bf2654208cd1c96ea7c0421d20fd35f3946d63f4a8a00a669b310d3d8a2: Status 404 returned error can't find the container with id fd780bf2654208cd1c96ea7c0421d20fd35f3946d63f4a8a00a669b310d3d8a2 Jul 22 18:39:52.492364 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:52.492342 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf45312_be61_4ecc_bf68_ee249f228c2b.slice/crio-dbbc105b4ca4f87784d1d63dce4438a9d40a60fc5ba071c1a8296de2f4a0948c WatchSource:0}: Error finding container dbbc105b4ca4f87784d1d63dce4438a9d40a60fc5ba071c1a8296de2f4a0948c: Status 404 returned error can't find the container with id dbbc105b4ca4f87784d1d63dce4438a9d40a60fc5ba071c1a8296de2f4a0948c Jul 22 18:39:52.494952 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:52.494650 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a193b7_d3c9_451c_af5c_fe08756dba3f.slice/crio-2f8d306a0c85941033397320ed7a563e4bc36a618cc3bb21df663f94d03365c1 WatchSource:0}: Error finding container 2f8d306a0c85941033397320ed7a563e4bc36a618cc3bb21df663f94d03365c1: Status 404 returned error can't find the container with id 2f8d306a0c85941033397320ed7a563e4bc36a618cc3bb21df663f94d03365c1 Jul 22 18:39:52.494952 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:52.494931 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87719eb_a436_4abc_99f3_db0c8db21201.slice/crio-fe5678af8345e3af98b002495d67736e46857d3afa92b7965d714a20c73bcfb8 WatchSource:0}: Error finding container fe5678af8345e3af98b002495d67736e46857d3afa92b7965d714a20c73bcfb8: Status 404 returned error can't find the container with id fe5678af8345e3af98b002495d67736e46857d3afa92b7965d714a20c73bcfb8 Jul 22 18:39:52.497131 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:52.497100 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode953389b_d0c5_4bc7_907a_1687430ff7ba.slice/crio-423a16f0ae8ecd8fae79af56dbc982eb9b18b20f218139505681e15f769500c0 WatchSource:0}: Error finding container 423a16f0ae8ecd8fae79af56dbc982eb9b18b20f218139505681e15f769500c0: Status 404 returned error can't find the container with id 423a16f0ae8ecd8fae79af56dbc982eb9b18b20f218139505681e15f769500c0 Jul 22 18:39:52.497672 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:52.497646 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e9199d_7044_4862_a13f_0cc2ef4ee32a.slice/crio-bf0b8e69fd49860eed2ce97b2f225579ce64a26be866c57518f87bafaf1b37e0 WatchSource:0}: Error finding container bf0b8e69fd49860eed2ce97b2f225579ce64a26be866c57518f87bafaf1b37e0: Status 404 returned error can't find the container with id bf0b8e69fd49860eed2ce97b2f225579ce64a26be866c57518f87bafaf1b37e0 Jul 22 18:39:52.523252 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.523231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:39:52.523383 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.523367 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:39:52.523439 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.523417 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs podName:d27e81e3-a9d6-4c4b-b138-3a4394204eb3 nodeName:}" failed. No retries permitted until 2026-07-22 18:39:53.523402731 +0000 UTC m=+3.970797946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs") pod "network-metrics-daemon-lc7qt" (UID: "d27e81e3-a9d6-4c4b-b138-3a4394204eb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:39:52.537891 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:39:52.537869 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5982cdd9aa07454c3773d04a38a8a229.slice/crio-c27bcd702b1d2cc3f58724809c8a06be5d2afd28853736735cae7824c79b2cad WatchSource:0}: Error finding container c27bcd702b1d2cc3f58724809c8a06be5d2afd28853736735cae7824c79b2cad: Status 404 returned error can't find the container with id c27bcd702b1d2cc3f58724809c8a06be5d2afd28853736735cae7824c79b2cad Jul 22 18:39:52.624260 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.624226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:52.624432 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.624394 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:39:52.624500 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.624476 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret podName:0f37a932-854e-4799-894e-db1e2b8a27d3 nodeName:}" failed. No retries permitted until 2026-07-22 18:39:53.624455002 +0000 UTC m=+4.071850231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret") pod "global-pull-secret-syncer-9xjws" (UID: "0f37a932-854e-4799-894e-db1e2b8a27d3") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:39:52.668524 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.668498 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 18:39:52.724621 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.724599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgl5v\" (UniqueName: \"kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v\") pod \"network-check-target-cvmqq\" (UID: \"cee0ab35-2e33-4a5d-83fe-c08b51c0ff47\") " pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:39:52.724735 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.724718 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:39:52.724735 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.724731 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:39:52.724856 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.724740 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mgl5v for pod openshift-network-diagnostics/network-check-target-cvmqq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:39:52.724856 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:52.724787 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v podName:cee0ab35-2e33-4a5d-83fe-c08b51c0ff47 nodeName:}" failed. No retries permitted until 2026-07-22 18:39:53.724775191 +0000 UTC m=+4.172170402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mgl5v" (UniqueName: "kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v") pod "network-check-target-cvmqq" (UID: "cee0ab35-2e33-4a5d-83fe-c08b51c0ff47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:39:52.904220 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.904160 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 18:34:51 +0000 UTC" deadline="2028-04-01 02:08:32.10189458 +0000 UTC" Jul 22 18:39:52.904220 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:52.904212 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14839h28m39.19768627s" Jul 22 18:39:53.000709 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.000155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:39:53.000709 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:53.000304 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:39:53.013700 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.013661 2574 generic.go:356] "Generic (PLEG): container finished" podID="749a592c7f4bbbb81e4b1740a322d6c0" containerID="b7015d65c5c989c6f2931013ea03adbcac7ce579dcbd95f8e3af4558fe1755b1" exitCode=0 Jul 22 18:39:53.013869 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.013754 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" event={"ID":"749a592c7f4bbbb81e4b1740a322d6c0","Type":"ContainerDied","Data":"b7015d65c5c989c6f2931013ea03adbcac7ce579dcbd95f8e3af4558fe1755b1"} Jul 22 18:39:53.018755 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.018678 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zt4fh" event={"ID":"e953389b-d0c5-4bc7-907a-1687430ff7ba","Type":"ContainerStarted","Data":"423a16f0ae8ecd8fae79af56dbc982eb9b18b20f218139505681e15f769500c0"} Jul 22 18:39:53.021727 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.021665 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" event={"ID":"e9e9199d-7044-4862-a13f-0cc2ef4ee32a","Type":"ContainerStarted","Data":"bf0b8e69fd49860eed2ce97b2f225579ce64a26be866c57518f87bafaf1b37e0"} Jul 22 18:39:53.026803 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.026704 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" event={"ID":"a87719eb-a436-4abc-99f3-db0c8db21201","Type":"ContainerStarted","Data":"fe5678af8345e3af98b002495d67736e46857d3afa92b7965d714a20c73bcfb8"} Jul 22 18:39:53.035246 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.034288 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ztgb" event={"ID":"73a193b7-d3c9-451c-af5c-fe08756dba3f","Type":"ContainerStarted","Data":"2f8d306a0c85941033397320ed7a563e4bc36a618cc3bb21df663f94d03365c1"} Jul 22 18:39:53.040193 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.039872 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7kgx" event={"ID":"daf45312-be61-4ecc-bf68-ee249f228c2b","Type":"ContainerStarted","Data":"dbbc105b4ca4f87784d1d63dce4438a9d40a60fc5ba071c1a8296de2f4a0948c"} Jul 22 18:39:53.047463 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.047213 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fw9m4" event={"ID":"f12f5040-f33c-4fbe-87a5-f95391205cd5","Type":"ContainerStarted","Data":"fd780bf2654208cd1c96ea7c0421d20fd35f3946d63f4a8a00a669b310d3d8a2"} Jul 22 18:39:53.051486 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.051424 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8n5kx" event={"ID":"64894f60-a8ee-4da7-a605-e109bb3a84aa","Type":"ContainerStarted","Data":"c2ac8de155701de97dd913ee50b8501059a9eb0dff645e49232854f3aa9f20ea"} Jul 22 18:39:53.052825 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.052773 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" event={"ID":"5982cdd9aa07454c3773d04a38a8a229","Type":"ContainerStarted","Data":"c27bcd702b1d2cc3f58724809c8a06be5d2afd28853736735cae7824c79b2cad"} Jul 22 18:39:53.056558 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.056489 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" event={"ID":"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d","Type":"ContainerStarted","Data":"01a3f50b7a008f95c0a6207faf99f3b039a71bc5189f628bc4d8a887f3a292ae"} Jul 22 18:39:53.058546 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.058522 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gg7vx" event={"ID":"4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4","Type":"ContainerStarted","Data":"af9e9d72c165e1de71e21a6a3ab23ca924d4bb5555ea9f0d852b13f0d09b4f71"} Jul 22 18:39:53.531142 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.531100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:39:53.531787 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:53.531258 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:39:53.531787 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:53.531321 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs podName:d27e81e3-a9d6-4c4b-b138-3a4394204eb3 nodeName:}" failed. No retries permitted until 2026-07-22 18:39:55.531304008 +0000 UTC m=+5.978699232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs") pod "network-metrics-daemon-lc7qt" (UID: "d27e81e3-a9d6-4c4b-b138-3a4394204eb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:39:53.632179 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.632120 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:53.632347 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:53.632273 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:39:53.632430 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:53.632347 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret podName:0f37a932-854e-4799-894e-db1e2b8a27d3 nodeName:}" failed. No retries permitted until 2026-07-22 18:39:55.63232595 +0000 UTC m=+6.079721172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret") pod "global-pull-secret-syncer-9xjws" (UID: "0f37a932-854e-4799-894e-db1e2b8a27d3") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:39:53.732833 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.732794 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgl5v\" (UniqueName: \"kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v\") pod \"network-check-target-cvmqq\" (UID: \"cee0ab35-2e33-4a5d-83fe-c08b51c0ff47\") " pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:39:53.733089 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:53.733001 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:39:53.733089 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:53.733024 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:39:53.733089 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:53.733037 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mgl5v for pod openshift-network-diagnostics/network-check-target-cvmqq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:39:53.733249 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:53.733093 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v podName:cee0ab35-2e33-4a5d-83fe-c08b51c0ff47 nodeName:}" failed. No retries permitted until 2026-07-22 18:39:55.733075979 +0000 UTC m=+6.180471193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mgl5v" (UniqueName: "kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v") pod "network-check-target-cvmqq" (UID: "cee0ab35-2e33-4a5d-83fe-c08b51c0ff47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:39:53.904881 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.904788 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 18:34:51 +0000 UTC" deadline="2028-01-18 07:22:11.302866659 +0000 UTC" Jul 22 18:39:53.904881 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:53.904832 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13068h42m17.39803922s" Jul 22 18:39:54.002552 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:54.002087 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:54.002552 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:54.002217 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:39:54.002552 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:54.002087 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:39:54.002552 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:54.002367 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:39:54.078943 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:54.078903 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" event={"ID":"749a592c7f4bbbb81e4b1740a322d6c0","Type":"ContainerStarted","Data":"8cd9c990ef6e4b9aa724c402b89766fd88a4109b9a7bbf3b8b7069902b460f27"} Jul 22 18:39:54.099617 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:54.099553 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" podStartSLOduration=3.099523519 podStartE2EDuration="3.099523519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:39:54.098988686 +0000 UTC m=+4.546383918" watchObservedRunningTime="2026-07-22 18:39:54.099523519 +0000 UTC m=+4.546918751" Jul 22 18:39:54.999860 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:54.999290 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:39:54.999860 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:54.999424 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:39:55.551077 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:55.550847 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:39:55.551916 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:55.551445 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:39:55.551916 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:55.551519 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs podName:d27e81e3-a9d6-4c4b-b138-3a4394204eb3 nodeName:}" failed. No retries permitted until 2026-07-22 18:39:59.551499013 +0000 UTC m=+9.998894237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs") pod "network-metrics-daemon-lc7qt" (UID: "d27e81e3-a9d6-4c4b-b138-3a4394204eb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:39:55.651938 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:55.651895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:55.652108 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:55.652060 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:39:55.652184 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:55.652123 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret podName:0f37a932-854e-4799-894e-db1e2b8a27d3 nodeName:}" failed. No retries permitted until 2026-07-22 18:39:59.652104216 +0000 UTC m=+10.099499433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret") pod "global-pull-secret-syncer-9xjws" (UID: "0f37a932-854e-4799-894e-db1e2b8a27d3") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:39:55.753750 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:55.753168 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgl5v\" (UniqueName: \"kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v\") pod \"network-check-target-cvmqq\" (UID: \"cee0ab35-2e33-4a5d-83fe-c08b51c0ff47\") " pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:39:55.753750 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:55.753322 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:39:55.753750 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:55.753344 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:39:55.753750 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:55.753357 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mgl5v for pod openshift-network-diagnostics/network-check-target-cvmqq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:39:55.753750 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:55.753412 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v podName:cee0ab35-2e33-4a5d-83fe-c08b51c0ff47 nodeName:}" failed. No retries permitted until 2026-07-22 18:39:59.753394439 +0000 UTC m=+10.200789656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mgl5v" (UniqueName: "kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v") pod "network-check-target-cvmqq" (UID: "cee0ab35-2e33-4a5d-83fe-c08b51c0ff47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:39:56.003175 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:56.002380 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:39:56.003175 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:56.002533 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:39:56.003175 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:56.002907 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:56.003175 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:56.003021 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:39:56.999879 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:56.999800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:39:57.000057 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:56.999941 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:39:58.001454 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:58.000910 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:58.001454 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:58.001062 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:39:58.001454 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:58.001450 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:39:58.002000 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:58.001559 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:39:58.999512 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:58.999433 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:39:58.999665 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:58.999566 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:39:59.584286 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:59.584251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:39:59.584746 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:59.584428 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:39:59.584746 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:59.584505 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs podName:d27e81e3-a9d6-4c4b-b138-3a4394204eb3 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:07.584484315 +0000 UTC m=+18.031879527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs") pod "network-metrics-daemon-lc7qt" (UID: "d27e81e3-a9d6-4c4b-b138-3a4394204eb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:39:59.685258 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:59.685227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:39:59.685425 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:59.685365 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:39:59.685477 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:59.685432 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret podName:0f37a932-854e-4799-894e-db1e2b8a27d3 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:07.685413829 +0000 UTC m=+18.132809040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret") pod "global-pull-secret-syncer-9xjws" (UID: "0f37a932-854e-4799-894e-db1e2b8a27d3") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:39:59.786618 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:59.786029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgl5v\" (UniqueName: \"kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v\") pod \"network-check-target-cvmqq\" (UID: \"cee0ab35-2e33-4a5d-83fe-c08b51c0ff47\") " pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:39:59.786618 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:59.786187 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:39:59.786618 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:59.786210 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:39:59.786618 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:59.786223 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mgl5v for pod openshift-network-diagnostics/network-check-target-cvmqq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:39:59.786618 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:39:59.786286 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v podName:cee0ab35-2e33-4a5d-83fe-c08b51c0ff47 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:07.786267196 +0000 UTC m=+18.233662417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mgl5v" (UniqueName: "kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v") pod "network-check-target-cvmqq" (UID: "cee0ab35-2e33-4a5d-83fe-c08b51c0ff47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:40:00.000317 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:39:59.999999 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:00.000317 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:00.000127 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:00.000317 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:00.000207 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:00.000317 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:00.000119 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:00.999934 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:00.999900 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:01.000411 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:01.000050 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:01.999505 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:01.999464 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:01.999673 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:01.999609 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:01.999902 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:01.999882 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:02.000105 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:02.000030 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:03.000069 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:03.000034 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:03.000207 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:03.000137 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:03.999166 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:03.999134 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:03.999305 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:03.999247 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:03.999360 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:03.999305 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:03.999411 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:03.999394 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:04.999168 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:04.999132 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:04.999620 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:04.999243 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:05.999619 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:05.999579 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:06.000081 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:05.999719 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:06.000081 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:05.999801 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:06.000081 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:05.999929 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:07.000007 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:06.999963 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:07.000430 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:07.000079 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:07.649428 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:07.649391 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:07.649612 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:07.649525 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:40:07.649612 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:07.649596 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs podName:d27e81e3-a9d6-4c4b-b138-3a4394204eb3 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:23.649576787 +0000 UTC m=+34.096972011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs") pod "network-metrics-daemon-lc7qt" (UID: "d27e81e3-a9d6-4c4b-b138-3a4394204eb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:40:07.750445 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:07.750408 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:07.750619 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:07.750573 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:40:07.750671 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:07.750647 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret podName:0f37a932-854e-4799-894e-db1e2b8a27d3 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:23.750627021 +0000 UTC m=+34.198022260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret") pod "global-pull-secret-syncer-9xjws" (UID: "0f37a932-854e-4799-894e-db1e2b8a27d3") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:40:07.851183 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:07.851147 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgl5v\" (UniqueName: \"kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v\") pod \"network-check-target-cvmqq\" (UID: \"cee0ab35-2e33-4a5d-83fe-c08b51c0ff47\") " pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:07.851397 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:07.851295 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:40:07.851397 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:07.851313 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:40:07.851397 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:07.851324 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mgl5v for pod openshift-network-diagnostics/network-check-target-cvmqq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:40:07.851397 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:07.851371 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v podName:cee0ab35-2e33-4a5d-83fe-c08b51c0ff47 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:23.851357123 +0000 UTC m=+34.298752333 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mgl5v" (UniqueName: "kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v") pod "network-check-target-cvmqq" (UID: "cee0ab35-2e33-4a5d-83fe-c08b51c0ff47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:40:07.999215 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:07.999136 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:07.999375 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:07.999279 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:07.999375 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:07.999326 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:07.999490 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:07.999428 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:08.999858 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:08.999812 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:09.000389 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:08.999946 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:10.000226 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:10.000192 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:10.000639 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:10.000302 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:10.000639 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:10.000368 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:10.000639 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:10.000482 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:10.999785 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:10.999748 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:11.000017 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:10.999888 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:11.999898 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:11.999867 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:12.000253 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:11.999985 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:12.000253 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:12.000042 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:12.000253 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:12.000134 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:13.000489 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.000051 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:13.001139 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:13.000593 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:13.121688 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.121621 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" event={"ID":"a87719eb-a436-4abc-99f3-db0c8db21201","Type":"ContainerStarted","Data":"074a0c87eb5674b61f72d1520b94721d868534d6c97eb4aed950a025f3b9fc10"} Jul 22 18:40:13.121816 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.121695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" event={"ID":"a87719eb-a436-4abc-99f3-db0c8db21201","Type":"ContainerStarted","Data":"4b0bed10624b0135ee5e5fa50a62ec3eebccb1f10d8826d702721129645c2ebd"} Jul 22 18:40:13.121816 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.121747 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" event={"ID":"a87719eb-a436-4abc-99f3-db0c8db21201","Type":"ContainerStarted","Data":"2ea2156dd1f9736d074a775bcb19481112b9e94da524913b11d75217e35cdb0d"} Jul 22 18:40:13.121816 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.121778 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" event={"ID":"a87719eb-a436-4abc-99f3-db0c8db21201","Type":"ContainerStarted","Data":"eb242ecf7824b25deb4fa134b95feb30bccc69563ab48476f1c081c9891af2d6"} Jul 22 18:40:13.121816 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.121814 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" event={"ID":"a87719eb-a436-4abc-99f3-db0c8db21201","Type":"ContainerStarted","Data":"4e77a8b22cf36e4b8ca23c456cbec585de44b7dccc43be2c80c7515ffa71a0cb"} Jul 22 18:40:13.122015 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.121855 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" event={"ID":"a87719eb-a436-4abc-99f3-db0c8db21201","Type":"ContainerStarted","Data":"ce8c70cbfd052e610b8307259d460349497b23352d4bc70954edd33cfa325a58"} Jul 22 18:40:13.122912 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.122882 2574 generic.go:356] "Generic (PLEG): container finished" podID="73a193b7-d3c9-451c-af5c-fe08756dba3f" containerID="d01c34a01ad15bd7c74b7afe05553ee976fd850262727a63491a3ae9b70f7bfe" exitCode=0 Jul 22 18:40:13.123040 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.122954 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ztgb" event={"ID":"73a193b7-d3c9-451c-af5c-fe08756dba3f","Type":"ContainerDied","Data":"d01c34a01ad15bd7c74b7afe05553ee976fd850262727a63491a3ae9b70f7bfe"} Jul 22 18:40:13.124439 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.124337 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7kgx" event={"ID":"daf45312-be61-4ecc-bf68-ee249f228c2b","Type":"ContainerStarted","Data":"df4d99c9a6efe5246e3a4cab26155bea021db37eacac9e2155f1db7354668289"} Jul 22 18:40:13.125655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.125635 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8n5kx" event={"ID":"64894f60-a8ee-4da7-a605-e109bb3a84aa","Type":"ContainerStarted","Data":"2794512d5bcedec28a94f3446de848ce51f1c5f4c8c91a1a90fa0519c4693df7"} Jul 22 18:40:13.127102 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.127076 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" event={"ID":"5982cdd9aa07454c3773d04a38a8a229","Type":"ContainerStarted","Data":"e7d8926214a19f43790f0883bde8c098cf6b55082e239a8f85a0a7619873bacb"} Jul 22 18:40:13.128945 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.128924 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" event={"ID":"96cfe8d7-79cd-41ca-8f7b-10e116d5a85d","Type":"ContainerStarted","Data":"668eb05638f69cb2e72d72294f458e1f03eeb63f00277b0577732da60d864d61"} Jul 22 18:40:13.130220 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.130203 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gg7vx" event={"ID":"4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4","Type":"ContainerStarted","Data":"d48d91f46bdf29049bbcafae42971bebd92e2eebc447e8e1809c00c75218795d"} Jul 22 18:40:13.131589 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.131570 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zt4fh" event={"ID":"e953389b-d0c5-4bc7-907a-1687430ff7ba","Type":"ContainerStarted","Data":"deaaddfe724139ec66616de3977a7bbba4d817dde98dc73528fdbea6094e4158"} Jul 22 18:40:13.132933 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.132914 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" event={"ID":"e9e9199d-7044-4862-a13f-0cc2ef4ee32a","Type":"ContainerStarted","Data":"0e51413e9a5991a5b8258c5d75c4180a19f9846bea475df971fec0a5234bd745"} Jul 22 18:40:13.154449 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.154411 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" podStartSLOduration=22.154399096 podStartE2EDuration="22.154399096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:40:13.154343318 +0000 UTC m=+23.601738556" watchObservedRunningTime="2026-07-22 18:40:13.154399096 +0000 UTC m=+23.601794327" Jul 22 18:40:13.168490 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.168446 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vtc9x" podStartSLOduration=3.364807391 podStartE2EDuration="23.168428802s" totalImagesPullingTime="19.803621411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:40:13.168243753 +0000 UTC m=+23.615638986" watchObservedRunningTime="2026-07-22 18:40:13.168428802 +0000 UTC m=+23.615824034" Jul 22 18:40:13.180986 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.180938 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j7kgx" podStartSLOduration=3.385161699 podStartE2EDuration="23.18092353s" totalImagesPullingTime="19.795761831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:40:13.180627932 +0000 UTC m=+23.628023165" watchObservedRunningTime="2026-07-22 18:40:13.18092353 +0000 UTC m=+23.628318765" Jul 22 18:40:13.198605 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.198561 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gg7vx" podStartSLOduration=3.445255126 podStartE2EDuration="23.198550151s" totalImagesPullingTime="19.753295025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:40:13.198355425 +0000 UTC m=+23.645750656" watchObservedRunningTime="2026-07-22 18:40:13.198550151 +0000 UTC m=+23.645945382" Jul 22 18:40:13.213946 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.213907 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zt4fh" podStartSLOduration=3.352854501 podStartE2EDuration="23.213895908s" totalImagesPullingTime="19.861041407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:40:13.213301708 +0000 UTC m=+23.660696941" watchObservedRunningTime="2026-07-22 18:40:13.213895908 +0000 UTC m=+23.661291142" Jul 22 18:40:13.877186 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.877148 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 18:40:13.951646 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.951555 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T18:40:13.8771681Z","UUID":"3dcdbc04-c3f5-4133-bf55-768559462e52","Handler":null,"Name":"","Endpoint":""} Jul 22 18:40:13.953066 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.953048 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 18:40:13.953172 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.953089 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 18:40:13.999397 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.999365 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:13.999397 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:13.999365 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:13.999637 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:13.999481 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:13.999867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:13.999843 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:14.136452 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:14.136363 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fw9m4" event={"ID":"f12f5040-f33c-4fbe-87a5-f95391205cd5","Type":"ContainerStarted","Data":"3229878b25083c356c66ccc81cbcbf17aa8be231ca580078b900a9656e64fcf9"} Jul 22 18:40:14.138519 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:14.138493 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" event={"ID":"e9e9199d-7044-4862-a13f-0cc2ef4ee32a","Type":"ContainerStarted","Data":"e180fdb53e320b2962880771eb8d6efee3ae37566e1e171f0e3ee99b8591db75"} Jul 22 18:40:14.150266 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:14.150228 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8n5kx" podStartSLOduration=12.838749551 podStartE2EDuration="24.150214011s" totalImagesPullingTime="11.31146446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:40:13.22957628 +0000 UTC m=+23.676971524" watchObservedRunningTime="2026-07-22 18:40:14.150214011 +0000 UTC m=+24.597609237" Jul 22 18:40:14.150902 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:14.150863 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fw9m4" podStartSLOduration=4.402863202 podStartE2EDuration="24.150846795s" totalImagesPullingTime="19.747983593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:40:14.150353762 +0000 UTC m=+24.597748988" watchObservedRunningTime="2026-07-22 18:40:14.150846795 +0000 UTC m=+24.598242028" Jul 22 18:40:14.151250 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:14.151230 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8n5kx" Jul 22 18:40:14.151750 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:14.151731 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8n5kx" Jul 22 18:40:15.000071 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:15.000037 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:15.000318 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:15.000168 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:15.999584 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:15.999407 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:15.999584 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:15.999407 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:16.000081 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:15.999709 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:16.000081 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:15.999829 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:16.144514 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:16.144474 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" event={"ID":"e9e9199d-7044-4862-a13f-0cc2ef4ee32a","Type":"ContainerStarted","Data":"e2fa1f7535bc79315d77dfbfaec44e50f68ccf4a3612167da23bde57aeeb0b73"} Jul 22 18:40:16.148030 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:16.148010 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 18:40:16.148160 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:16.148009 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" event={"ID":"a87719eb-a436-4abc-99f3-db0c8db21201","Type":"ContainerStarted","Data":"360c0d8516aafd154199f4dfb54adf63918c7c6b51943cb0d536ed47b6c0c3e2"} Jul 22 18:40:16.999527 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:16.999496 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:16.999771 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:16.999631 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:17.999559 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:17.999372 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:17.999636 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:17.999410 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:17.999695 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:17.999674 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:17.999851 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:17.999821 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:18.155993 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:18.155902 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" event={"ID":"a87719eb-a436-4abc-99f3-db0c8db21201","Type":"ContainerStarted","Data":"88f9a9274a1214c9daeaa4851ba8b4ce427271393bdcd8e0eddda4a739565905"} Jul 22 18:40:18.156288 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:18.156263 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:40:18.156376 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:18.156348 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:40:18.158124 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:18.158098 2574 generic.go:356] "Generic (PLEG): container finished" podID="73a193b7-d3c9-451c-af5c-fe08756dba3f" containerID="79e2690b466471adfd21b99f97f7c47c4b813644d4f0e9c9a4138195c12daf96" exitCode=0 Jul 22 18:40:18.158233 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:18.158149 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ztgb" event={"ID":"73a193b7-d3c9-451c-af5c-fe08756dba3f","Type":"ContainerDied","Data":"79e2690b466471adfd21b99f97f7c47c4b813644d4f0e9c9a4138195c12daf96"} Jul 22 18:40:18.172394 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:18.172370 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:40:18.179471 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:18.179432 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" podStartSLOduration=8.386998418 podStartE2EDuration="28.17941898s" totalImagesPullingTime="19.792420562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:40:18.178104783 +0000 UTC m=+28.625500015" watchObservedRunningTime="2026-07-22 18:40:18.17941898 +0000 UTC m=+28.626814211" Jul 22 18:40:18.179661 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:18.179642 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9s2q" podStartSLOduration=6.007674586 podStartE2EDuration="28.17963534s" totalImagesPullingTime="22.171960754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:39:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:40:16.15945906 +0000 UTC m=+26.606854291" watchObservedRunningTime="2026-07-22 18:40:18.17963534 +0000 UTC m=+28.627030630" Jul 22 18:40:18.383374 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:18.383346 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8n5kx" Jul 22 18:40:18.383527 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:18.383465 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 18:40:18.383887 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:18.383869 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8n5kx" Jul 22 18:40:18.999735 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:18.999666 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:18.999881 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:18.999767 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:19.160566 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:19.160529 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:40:19.174630 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:19.174606 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:40:19.620514 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:19.620485 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9xjws"] Jul 22 18:40:19.620689 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:19.620591 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:19.620740 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:19.620681 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:19.623924 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:19.623899 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cvmqq"] Jul 22 18:40:19.624087 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:19.624011 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:19.624147 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:19.624109 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:19.624462 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:19.624442 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lc7qt"] Jul 22 18:40:19.624539 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:19.624527 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:19.624614 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:19.624597 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:20.999414 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:20.999390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:20.999414 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:20.999403 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:20.999765 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:20.999398 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:20.999765 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:20.999483 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:20.999765 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:20.999574 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:20.999765 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:20.999654 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:21.165885 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:21.165863 2574 generic.go:356] "Generic (PLEG): container finished" podID="73a193b7-d3c9-451c-af5c-fe08756dba3f" containerID="269014c9ca7deca72f1aa24355f0e78595520689198e4d12f47b801ec4cc382d" exitCode=0 Jul 22 18:40:21.166002 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:21.165947 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ztgb" event={"ID":"73a193b7-d3c9-451c-af5c-fe08756dba3f","Type":"ContainerDied","Data":"269014c9ca7deca72f1aa24355f0e78595520689198e4d12f47b801ec4cc382d"} Jul 22 18:40:22.170372 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:22.170143 2574 generic.go:356] "Generic (PLEG): container finished" podID="73a193b7-d3c9-451c-af5c-fe08756dba3f" containerID="6ae77c91ba8c3994aecbc1b32ca025f2d4fc402ae373d811081db60e03deb19a" exitCode=0 Jul 22 18:40:22.170372 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:22.170217 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ztgb" event={"ID":"73a193b7-d3c9-451c-af5c-fe08756dba3f","Type":"ContainerDied","Data":"6ae77c91ba8c3994aecbc1b32ca025f2d4fc402ae373d811081db60e03deb19a"} Jul 22 18:40:22.999733 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:22.999696 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:22.999908 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:22.999739 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:22.999908 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:22.999739 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:22.999908 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:22.999886 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:23.006257 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:23.006229 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:23.006368 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:23.006314 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:23.673312 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:23.673283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:23.673661 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:23.673400 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:40:23.673661 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:23.673450 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs podName:d27e81e3-a9d6-4c4b-b138-3a4394204eb3 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:55.673437315 +0000 UTC m=+66.120832525 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs") pod "network-metrics-daemon-lc7qt" (UID: "d27e81e3-a9d6-4c4b-b138-3a4394204eb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:40:23.774190 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:23.774157 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:23.774345 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:23.774303 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:40:23.774409 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:23.774384 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret podName:0f37a932-854e-4799-894e-db1e2b8a27d3 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:55.774362145 +0000 UTC m=+66.221757374 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret") pod "global-pull-secret-syncer-9xjws" (UID: "0f37a932-854e-4799-894e-db1e2b8a27d3") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:40:23.875033 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:23.874998 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgl5v\" (UniqueName: \"kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v\") pod \"network-check-target-cvmqq\" (UID: \"cee0ab35-2e33-4a5d-83fe-c08b51c0ff47\") " pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:23.875205 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:23.875148 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:40:23.875205 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:23.875167 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:40:23.875205 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:23.875179 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mgl5v for pod openshift-network-diagnostics/network-check-target-cvmqq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:40:23.875379 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:23.875237 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v podName:cee0ab35-2e33-4a5d-83fe-c08b51c0ff47 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:55.875219993 +0000 UTC m=+66.322615209 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mgl5v" (UniqueName: "kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v") pod "network-check-target-cvmqq" (UID: "cee0ab35-2e33-4a5d-83fe-c08b51c0ff47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:40:24.999291 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:24.999258 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:24.999291 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:24.999300 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:24.999763 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:24.999262 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:24.999763 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:24.999395 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:40:24.999763 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:24.999483 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xjws" podUID="0f37a932-854e-4799-894e-db1e2b8a27d3" Jul 22 18:40:24.999763 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:24.999541 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvmqq" podUID="cee0ab35-2e33-4a5d-83fe-c08b51c0ff47" Jul 22 18:40:25.869595 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.869565 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeReady" Jul 22 18:40:25.869738 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.869711 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 18:40:25.930192 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.930144 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-97cd48d98-2wkxk"] Jul 22 18:40:25.935275 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.935251 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:25.937617 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.937592 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 18:40:25.937750 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.937693 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bzh2s\"" Jul 22 18:40:25.937750 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.937717 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 18:40:25.937750 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.937732 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 18:40:25.947138 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.946884 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 18:40:25.948907 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.948886 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-97cd48d98-2wkxk"] Jul 22 18:40:25.951515 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.951490 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8ph6g"] Jul 22 18:40:25.954951 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.954931 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8"] Jul 22 18:40:25.955093 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.955072 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8ph6g" Jul 22 18:40:25.957637 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.957520 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 18:40:25.957637 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.957519 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 18:40:25.957801 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.957742 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zqsx4\"" Jul 22 18:40:25.958086 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.958065 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 18:40:25.958184 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.958102 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x262w"] Jul 22 18:40:25.958241 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.958226 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:40:25.960200 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.960181 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7492w\"" Jul 22 18:40:25.960289 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.960204 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 22 18:40:25.960289 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.960189 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 22 18:40:25.961404 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.961387 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x262w" Jul 22 18:40:25.963334 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.963316 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 18:40:25.963428 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.963388 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 18:40:25.963490 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.963391 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hbkg4\"" Jul 22 18:40:25.966442 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.966422 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8ph6g"] Jul 22 18:40:25.971575 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.971557 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x262w"] Jul 22 18:40:25.980206 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.980175 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8"] Jul 22 18:40:25.991189 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f689b8a-ab35-4f36-a20d-047e5614eac1-image-registry-private-configuration\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:25.991189 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991200 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:25.991379 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:40:25.991379 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cc3535-6546-4f2d-a502-9f3a97a63eaa-tmp-dir\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:25.991379 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991295 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b117ac7f-9bc8-46e2-a049-fdc145a52523-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:40:25.991379 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt85f\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-kube-api-access-mt85f\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:25.991585 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991401 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fe1adde-5816-463f-aa6a-7769d7569b7f-cert\") pod \"ingress-canary-8ph6g\" (UID: \"7fe1adde-5816-463f-aa6a-7769d7569b7f\") " pod="openshift-ingress-canary/ingress-canary-8ph6g" Jul 22 18:40:25.991585 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991471 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gltnj\" (UniqueName: \"kubernetes.io/projected/42cc3535-6546-4f2d-a502-9f3a97a63eaa-kube-api-access-gltnj\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:25.991585 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f689b8a-ab35-4f36-a20d-047e5614eac1-ca-trust-extracted\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:25.991585 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f689b8a-ab35-4f36-a20d-047e5614eac1-installation-pull-secrets\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:25.991585 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991552 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-certificates\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:25.991585 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42cc3535-6546-4f2d-a502-9f3a97a63eaa-config-volume\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:25.991849 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f689b8a-ab35-4f36-a20d-047e5614eac1-trusted-ca\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:25.991849 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991674 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-bound-sa-token\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:25.991849 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991714 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:25.991849 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:25.991746 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9mnk\" (UniqueName: \"kubernetes.io/projected/7fe1adde-5816-463f-aa6a-7769d7569b7f-kube-api-access-g9mnk\") pod \"ingress-canary-8ph6g\" (UID: \"7fe1adde-5816-463f-aa6a-7769d7569b7f\") " pod="openshift-ingress-canary/ingress-canary-8ph6g" Jul 22 18:40:26.093150 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093060 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gltnj\" (UniqueName: \"kubernetes.io/projected/42cc3535-6546-4f2d-a502-9f3a97a63eaa-kube-api-access-gltnj\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:26.093150 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f689b8a-ab35-4f36-a20d-047e5614eac1-ca-trust-extracted\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.093150 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f689b8a-ab35-4f36-a20d-047e5614eac1-installation-pull-secrets\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-certificates\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093434 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42cc3535-6546-4f2d-a502-9f3a97a63eaa-config-volume\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f689b8a-ab35-4f36-a20d-047e5614eac1-trusted-ca\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093527 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f689b8a-ab35-4f36-a20d-047e5614eac1-ca-trust-extracted\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-bound-sa-token\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mnk\" (UniqueName: \"kubernetes.io/projected/7fe1adde-5816-463f-aa6a-7769d7569b7f-kube-api-access-g9mnk\") pod \"ingress-canary-8ph6g\" (UID: \"7fe1adde-5816-463f-aa6a-7769d7569b7f\") " pod="openshift-ingress-canary/ingress-canary-8ph6g" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093625 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f689b8a-ab35-4f36-a20d-047e5614eac1-image-registry-private-configuration\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093651 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cc3535-6546-4f2d-a502-9f3a97a63eaa-tmp-dir\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093739 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b117ac7f-9bc8-46e2-a049-fdc145a52523-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093791 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-certificates\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.093838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt85f\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-kube-api-access-mt85f\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.094479 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.093843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fe1adde-5816-463f-aa6a-7769d7569b7f-cert\") pod \"ingress-canary-8ph6g\" (UID: \"7fe1adde-5816-463f-aa6a-7769d7569b7f\") " pod="openshift-ingress-canary/ingress-canary-8ph6g" Jul 22 18:40:26.094479 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.093917 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:40:26.094479 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.093936 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cd48d98-2wkxk: secret "image-registry-tls" not found Jul 22 18:40:26.094479 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.094019 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42cc3535-6546-4f2d-a502-9f3a97a63eaa-config-volume\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:26.094479 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.094030 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls podName:3f689b8a-ab35-4f36-a20d-047e5614eac1 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:26.594003014 +0000 UTC m=+37.041398252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls") pod "image-registry-97cd48d98-2wkxk" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1") : secret "image-registry-tls" not found Jul 22 18:40:26.094479 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.094098 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 18:40:26.094479 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.094154 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert podName:b117ac7f-9bc8-46e2-a049-fdc145a52523 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:26.594141418 +0000 UTC m=+37.041536646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-78xp8" (UID: "b117ac7f-9bc8-46e2-a049-fdc145a52523") : secret "networking-console-plugin-cert" not found Jul 22 18:40:26.094479 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.094173 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:40:26.094479 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.094239 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls podName:42cc3535-6546-4f2d-a502-9f3a97a63eaa nodeName:}" failed. No retries permitted until 2026-07-22 18:40:26.594220912 +0000 UTC m=+37.041616136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls") pod "dns-default-x262w" (UID: "42cc3535-6546-4f2d-a502-9f3a97a63eaa") : secret "dns-default-metrics-tls" not found Jul 22 18:40:26.094479 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.094281 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cc3535-6546-4f2d-a502-9f3a97a63eaa-tmp-dir\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:26.094479 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.094439 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f689b8a-ab35-4f36-a20d-047e5614eac1-trusted-ca\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.094959 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.094810 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b117ac7f-9bc8-46e2-a049-fdc145a52523-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:40:26.098296 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.098270 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f689b8a-ab35-4f36-a20d-047e5614eac1-installation-pull-secrets\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.098397 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.098298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fe1adde-5816-463f-aa6a-7769d7569b7f-cert\") pod \"ingress-canary-8ph6g\" (UID: \"7fe1adde-5816-463f-aa6a-7769d7569b7f\") " pod="openshift-ingress-canary/ingress-canary-8ph6g" Jul 22 18:40:26.098397 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.098306 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f689b8a-ab35-4f36-a20d-047e5614eac1-image-registry-private-configuration\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.102244 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.102223 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt85f\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-kube-api-access-mt85f\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.102344 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.102283 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gltnj\" (UniqueName: \"kubernetes.io/projected/42cc3535-6546-4f2d-a502-9f3a97a63eaa-kube-api-access-gltnj\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:26.103122 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.103100 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-bound-sa-token\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.103464 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.103425 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9mnk\" (UniqueName: \"kubernetes.io/projected/7fe1adde-5816-463f-aa6a-7769d7569b7f-kube-api-access-g9mnk\") pod \"ingress-canary-8ph6g\" (UID: \"7fe1adde-5816-463f-aa6a-7769d7569b7f\") " pod="openshift-ingress-canary/ingress-canary-8ph6g" Jul 22 18:40:26.266367 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.266335 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8ph6g" Jul 22 18:40:26.482354 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.482199 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8ph6g"] Jul 22 18:40:26.485580 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:40:26.485551 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe1adde_5816_463f_aa6a_7769d7569b7f.slice/crio-ff9c9520af07c4771c4352dbf037f583dcf38891cf82b3383d4c6372060fcdb9 WatchSource:0}: Error finding container ff9c9520af07c4771c4352dbf037f583dcf38891cf82b3383d4c6372060fcdb9: Status 404 returned error can't find the container with id ff9c9520af07c4771c4352dbf037f583dcf38891cf82b3383d4c6372060fcdb9 Jul 22 18:40:26.597449 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.597426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:26.597562 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.597459 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:26.597562 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.597479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:40:26.597666 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.597578 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 18:40:26.597666 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.597588 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:40:26.597666 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.597615 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:40:26.597666 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.597638 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cd48d98-2wkxk: secret "image-registry-tls" not found Jul 22 18:40:26.597810 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.597628 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert podName:b117ac7f-9bc8-46e2-a049-fdc145a52523 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:27.597615318 +0000 UTC m=+38.045010527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-78xp8" (UID: "b117ac7f-9bc8-46e2-a049-fdc145a52523") : secret "networking-console-plugin-cert" not found Jul 22 18:40:26.597810 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.597703 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls podName:42cc3535-6546-4f2d-a502-9f3a97a63eaa nodeName:}" failed. No retries permitted until 2026-07-22 18:40:27.597682461 +0000 UTC m=+38.045077671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls") pod "dns-default-x262w" (UID: "42cc3535-6546-4f2d-a502-9f3a97a63eaa") : secret "dns-default-metrics-tls" not found Jul 22 18:40:26.597810 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:26.597717 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls podName:3f689b8a-ab35-4f36-a20d-047e5614eac1 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:27.597710521 +0000 UTC m=+38.045105733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls") pod "image-registry-97cd48d98-2wkxk" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1") : secret "image-registry-tls" not found Jul 22 18:40:26.999377 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.999278 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:26.999377 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.999324 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:26.999581 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:26.999283 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:27.004581 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:27.002138 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 18:40:27.005101 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:27.005067 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 18:40:27.005229 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:27.005145 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-csrvs\"" Jul 22 18:40:27.005229 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:27.005145 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 18:40:27.005229 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:27.005162 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g2vpr\"" Jul 22 18:40:27.005229 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:27.005160 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 18:40:27.184775 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:27.184745 2574 generic.go:356] "Generic (PLEG): container finished" podID="73a193b7-d3c9-451c-af5c-fe08756dba3f" containerID="fc884c59ca80123ae86cc4132ea47161cf2772550e6c0c85ea66315568349d80" exitCode=0 Jul 22 18:40:27.185126 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:27.184839 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ztgb" event={"ID":"73a193b7-d3c9-451c-af5c-fe08756dba3f","Type":"ContainerDied","Data":"fc884c59ca80123ae86cc4132ea47161cf2772550e6c0c85ea66315568349d80"} Jul 22 18:40:27.186370 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:27.185931 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8ph6g" event={"ID":"7fe1adde-5816-463f-aa6a-7769d7569b7f","Type":"ContainerStarted","Data":"ff9c9520af07c4771c4352dbf037f583dcf38891cf82b3383d4c6372060fcdb9"} Jul 22 18:40:27.607471 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:27.607384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:27.607471 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:27.607442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:27.607711 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:27.607475 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:40:27.607711 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:27.607557 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:40:27.607711 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:27.607617 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 18:40:27.607711 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:27.607628 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls podName:42cc3535-6546-4f2d-a502-9f3a97a63eaa nodeName:}" failed. No retries permitted until 2026-07-22 18:40:29.607609016 +0000 UTC m=+40.055004230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls") pod "dns-default-x262w" (UID: "42cc3535-6546-4f2d-a502-9f3a97a63eaa") : secret "dns-default-metrics-tls" not found Jul 22 18:40:27.607711 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:27.607660 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert podName:b117ac7f-9bc8-46e2-a049-fdc145a52523 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:29.607646592 +0000 UTC m=+40.055041803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-78xp8" (UID: "b117ac7f-9bc8-46e2-a049-fdc145a52523") : secret "networking-console-plugin-cert" not found Jul 22 18:40:27.607711 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:27.607621 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:40:27.607711 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:27.607693 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cd48d98-2wkxk: secret "image-registry-tls" not found Jul 22 18:40:27.607995 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:27.607737 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls podName:3f689b8a-ab35-4f36-a20d-047e5614eac1 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:29.607723306 +0000 UTC m=+40.055118522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls") pod "image-registry-97cd48d98-2wkxk" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1") : secret "image-registry-tls" not found Jul 22 18:40:27.990880 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:27.990792 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:28.071910 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:28.071877 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:28.192454 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:28.192422 2574 generic.go:356] "Generic (PLEG): container finished" podID="73a193b7-d3c9-451c-af5c-fe08756dba3f" containerID="400bf1c1515dfbdc946cc3d6e0c8a91d64e5c244d9a1cd866e626e4819f4c5e5" exitCode=0 Jul 22 18:40:28.192902 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:28.192470 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ztgb" event={"ID":"73a193b7-d3c9-451c-af5c-fe08756dba3f","Type":"ContainerDied","Data":"400bf1c1515dfbdc946cc3d6e0c8a91d64e5c244d9a1cd866e626e4819f4c5e5"} Jul 22 18:40:29.199522 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:29.199273 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ztgb" event={"ID":"73a193b7-d3c9-451c-af5c-fe08756dba3f","Type":"ContainerStarted","Data":"0a9af9443b461ef4c4060f70bf4a35869e01792450a0dc4a4facdb972e336126"} Jul 22 18:40:29.226292 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:29.226224 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8ztgb" podStartSLOduration=7.031774203 podStartE2EDuration="39.22620338s" totalImagesPullingTime="32.194429177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:39:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:40:29.220453619 +0000 UTC m=+39.667848850" watchObservedRunningTime="2026-07-22 18:40:29.22620338 +0000 UTC m=+39.673598614" Jul 22 18:40:29.353085 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:29.353052 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:29.448533 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:29.448463 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:29.548134 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:29.548106 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:29.623089 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:29.623063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:29.623174 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:29.623098 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:29.623174 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:29.623116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:40:29.623249 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:29.623206 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:40:29.623249 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:29.623223 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 18:40:29.623249 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:29.623228 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:40:29.623249 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:29.623243 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cd48d98-2wkxk: secret "image-registry-tls" not found Jul 22 18:40:29.623357 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:29.623266 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls podName:42cc3535-6546-4f2d-a502-9f3a97a63eaa nodeName:}" failed. No retries permitted until 2026-07-22 18:40:33.623250822 +0000 UTC m=+44.070646032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls") pod "dns-default-x262w" (UID: "42cc3535-6546-4f2d-a502-9f3a97a63eaa") : secret "dns-default-metrics-tls" not found Jul 22 18:40:29.623357 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:29.623279 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert podName:b117ac7f-9bc8-46e2-a049-fdc145a52523 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:33.623272779 +0000 UTC m=+44.070667989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-78xp8" (UID: "b117ac7f-9bc8-46e2-a049-fdc145a52523") : secret "networking-console-plugin-cert" not found Jul 22 18:40:29.623357 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:29.623310 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls podName:3f689b8a-ab35-4f36-a20d-047e5614eac1 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:33.62329859 +0000 UTC m=+44.070693804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls") pod "image-registry-97cd48d98-2wkxk" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1") : secret "image-registry-tls" not found Jul 22 18:40:30.203214 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:30.203176 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8ph6g" event={"ID":"7fe1adde-5816-463f-aa6a-7769d7569b7f","Type":"ContainerStarted","Data":"3cf62d9d52a95739d6945b89e5e1bdfd6638961ac511da5d7ee88f964d288f97"} Jul 22 18:40:30.219702 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:30.219654 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8ph6g" podStartSLOduration=2.118196116 podStartE2EDuration="5.219639095s" totalImagesPullingTime="3.101442979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:40:30.219625943 +0000 UTC m=+40.667021183" watchObservedRunningTime="2026-07-22 18:40:30.219639095 +0000 UTC m=+40.667034321" Jul 22 18:40:31.459714 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:31.459682 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:31.511322 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:31.511289 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:31.713447 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:31.713372 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:31.789954 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:31.789924 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:33.652817 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:33.652785 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:33.653271 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:33.652824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:33.653271 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:33.652845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:40:33.653271 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:33.652937 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 18:40:33.653271 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:33.652963 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:40:33.653271 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:33.652989 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:40:33.653271 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:33.653004 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert podName:b117ac7f-9bc8-46e2-a049-fdc145a52523 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:41.652990934 +0000 UTC m=+52.100386145 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-78xp8" (UID: "b117ac7f-9bc8-46e2-a049-fdc145a52523") : secret "networking-console-plugin-cert" not found Jul 22 18:40:33.653271 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:33.653001 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cd48d98-2wkxk: secret "image-registry-tls" not found Jul 22 18:40:33.653271 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:33.653052 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls podName:42cc3535-6546-4f2d-a502-9f3a97a63eaa nodeName:}" failed. No retries permitted until 2026-07-22 18:40:41.653034909 +0000 UTC m=+52.100430122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls") pod "dns-default-x262w" (UID: "42cc3535-6546-4f2d-a502-9f3a97a63eaa") : secret "dns-default-metrics-tls" not found Jul 22 18:40:33.653271 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:33.653103 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls podName:3f689b8a-ab35-4f36-a20d-047e5614eac1 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:41.653088812 +0000 UTC m=+52.100484025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls") pod "image-registry-97cd48d98-2wkxk" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1") : secret "image-registry-tls" not found Jul 22 18:40:39.044881 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:39.044845 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:39.096435 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:39.096406 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:39.264652 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:39.264622 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:39.321641 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:39.321573 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:41.711472 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:41.711442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:41.711472 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:41.711477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:41.711867 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:41.711496 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:40:41.711867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:41.711580 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:40:41.711867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:41.711592 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:40:41.711867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:41.711614 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cd48d98-2wkxk: secret "image-registry-tls" not found Jul 22 18:40:41.711867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:41.711634 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls podName:42cc3535-6546-4f2d-a502-9f3a97a63eaa nodeName:}" failed. No retries permitted until 2026-07-22 18:40:57.711617987 +0000 UTC m=+68.159013198 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls") pod "dns-default-x262w" (UID: "42cc3535-6546-4f2d-a502-9f3a97a63eaa") : secret "dns-default-metrics-tls" not found Jul 22 18:40:41.711867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:41.711675 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls podName:3f689b8a-ab35-4f36-a20d-047e5614eac1 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:57.711662656 +0000 UTC m=+68.159057866 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls") pod "image-registry-97cd48d98-2wkxk" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1") : secret "image-registry-tls" not found Jul 22 18:40:41.711867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:41.711601 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 18:40:41.711867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:41.711721 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert podName:b117ac7f-9bc8-46e2-a049-fdc145a52523 nodeName:}" failed. No retries permitted until 2026-07-22 18:40:57.711709824 +0000 UTC m=+68.159105039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-78xp8" (UID: "b117ac7f-9bc8-46e2-a049-fdc145a52523") : secret "networking-console-plugin-cert" not found Jul 22 18:40:51.175844 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:51.175799 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s8l6" Jul 22 18:40:51.326956 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:51.326924 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:51.377850 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:51.377815 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:53.342769 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:53.342735 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:53.400220 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:53.400189 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:40:55.714915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:55.714867 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:40:55.717378 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:55.717360 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 18:40:55.725955 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:55.725936 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 18:40:55.726065 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:55.726010 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs podName:d27e81e3-a9d6-4c4b-b138-3a4394204eb3 nodeName:}" failed. No retries permitted until 2026-07-22 18:41:59.72599419 +0000 UTC m=+130.173389413 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs") pod "network-metrics-daemon-lc7qt" (UID: "d27e81e3-a9d6-4c4b-b138-3a4394204eb3") : secret "metrics-daemon-secret" not found Jul 22 18:40:55.815463 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:55.815435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:55.817562 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:55.817544 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 18:40:55.830108 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:55.830082 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f37a932-854e-4799-894e-db1e2b8a27d3-original-pull-secret\") pod \"global-pull-secret-syncer-9xjws\" (UID: \"0f37a932-854e-4799-894e-db1e2b8a27d3\") " pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:55.916608 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:55.916578 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgl5v\" (UniqueName: \"kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v\") pod \"network-check-target-cvmqq\" (UID: \"cee0ab35-2e33-4a5d-83fe-c08b51c0ff47\") " pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:55.919174 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:55.919154 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 18:40:55.928508 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:55.928488 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 18:40:55.939254 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:55.939234 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgl5v\" (UniqueName: \"kubernetes.io/projected/cee0ab35-2e33-4a5d-83fe-c08b51c0ff47-kube-api-access-mgl5v\") pod \"network-check-target-cvmqq\" (UID: \"cee0ab35-2e33-4a5d-83fe-c08b51c0ff47\") " pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:56.111856 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:56.111812 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xjws" Jul 22 18:40:56.119322 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:56.119296 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-csrvs\"" Jul 22 18:40:56.128174 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:56.128150 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:40:56.258511 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:56.258480 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9xjws"] Jul 22 18:40:56.261882 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:40:56.261848 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f37a932_854e_4799_894e_db1e2b8a27d3.slice/crio-b1d9febaf49a79ae88a4eb40b598242b784e5210ff90502345a507d5fec8e8ce WatchSource:0}: Error finding container b1d9febaf49a79ae88a4eb40b598242b784e5210ff90502345a507d5fec8e8ce: Status 404 returned error can't find the container with id b1d9febaf49a79ae88a4eb40b598242b784e5210ff90502345a507d5fec8e8ce Jul 22 18:40:56.269033 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:56.269011 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cvmqq"] Jul 22 18:40:56.271639 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:40:56.271616 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee0ab35_2e33_4a5d_83fe_c08b51c0ff47.slice/crio-2e5f8fed6da89077f3cd3cd56302a23ddf8dccfc02a47abda6177fc2fe0b7b8f WatchSource:0}: Error finding container 2e5f8fed6da89077f3cd3cd56302a23ddf8dccfc02a47abda6177fc2fe0b7b8f: Status 404 returned error can't find the container with id 2e5f8fed6da89077f3cd3cd56302a23ddf8dccfc02a47abda6177fc2fe0b7b8f Jul 22 18:40:57.266376 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:57.266332 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cvmqq" event={"ID":"cee0ab35-2e33-4a5d-83fe-c08b51c0ff47","Type":"ContainerStarted","Data":"2e5f8fed6da89077f3cd3cd56302a23ddf8dccfc02a47abda6177fc2fe0b7b8f"} Jul 22 18:40:57.267536 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:57.267501 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9xjws" event={"ID":"0f37a932-854e-4799-894e-db1e2b8a27d3","Type":"ContainerStarted","Data":"b1d9febaf49a79ae88a4eb40b598242b784e5210ff90502345a507d5fec8e8ce"} Jul 22 18:40:57.729528 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:57.729496 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:40:57.729704 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:57.729537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:40:57.729704 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:40:57.729560 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:40:57.729704 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:57.729660 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 18:40:57.729867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:57.729722 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert podName:b117ac7f-9bc8-46e2-a049-fdc145a52523 nodeName:}" failed. No retries permitted until 2026-07-22 18:41:29.729703342 +0000 UTC m=+100.177098556 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-78xp8" (UID: "b117ac7f-9bc8-46e2-a049-fdc145a52523") : secret "networking-console-plugin-cert" not found Jul 22 18:40:57.729867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:57.729662 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:40:57.729867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:57.729769 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cd48d98-2wkxk: secret "image-registry-tls" not found Jul 22 18:40:57.729867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:57.729662 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:40:57.729867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:57.729825 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls podName:3f689b8a-ab35-4f36-a20d-047e5614eac1 nodeName:}" failed. No retries permitted until 2026-07-22 18:41:29.729807929 +0000 UTC m=+100.177203141 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls") pod "image-registry-97cd48d98-2wkxk" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1") : secret "image-registry-tls" not found Jul 22 18:40:57.729867 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:40:57.729845 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls podName:42cc3535-6546-4f2d-a502-9f3a97a63eaa nodeName:}" failed. No retries permitted until 2026-07-22 18:41:29.72983553 +0000 UTC m=+100.177230739 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls") pod "dns-default-x262w" (UID: "42cc3535-6546-4f2d-a502-9f3a97a63eaa") : secret "dns-default-metrics-tls" not found Jul 22 18:41:00.276699 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:00.276656 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cvmqq" event={"ID":"cee0ab35-2e33-4a5d-83fe-c08b51c0ff47","Type":"ContainerStarted","Data":"fb035f9215c62f5a475709b1c5e7de43cb663c39cbdef51248cb810d11f1f149"} Jul 22 18:41:00.277220 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:00.276819 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:41:00.291302 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:00.291245 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cvmqq" podStartSLOduration=67.148094512 podStartE2EDuration="1m10.291228358s" totalImagesPullingTime="3.143133846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:41:00.289942268 +0000 UTC m=+70.737337502" watchObservedRunningTime="2026-07-22 18:41:00.291228358 +0000 UTC m=+70.738623589" Jul 22 18:41:00.980593 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:00.980553 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:41:01.080990 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:01.080942 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:41:03.285984 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:03.285932 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9xjws" event={"ID":"0f37a932-854e-4799-894e-db1e2b8a27d3","Type":"ContainerStarted","Data":"2d50a71c0b8ab2c206401b28c624b99c0dc2551c08bd91de070e5639320bc807"} Jul 22 18:41:03.299901 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:03.299856 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9xjws" podStartSLOduration=66.984327954 podStartE2EDuration="1m13.299841866s" totalImagesPullingTime="6.315513912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:41:03.299254935 +0000 UTC m=+73.746650163" watchObservedRunningTime="2026-07-22 18:41:03.299841866 +0000 UTC m=+73.747237076" Jul 22 18:41:06.076017 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:06.075967 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:41:06.129290 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:06.129263 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:41:17.737304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.737190 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8"] Jul 22 18:41:17.741274 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.741257 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:41:17.744141 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.744120 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 18:41:17.744276 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.744181 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 18:41:17.744276 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.744207 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 18:41:17.744276 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.744217 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 22 18:41:17.749516 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.749479 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8"] Jul 22 18:41:17.864762 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.864740 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d7170f4c-193a-4693-babf-1942283dbf38-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d5b44c959-ckdh8\" (UID: \"d7170f4c-193a-4693-babf-1942283dbf38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:41:17.864891 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.864770 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7170f4c-193a-4693-babf-1942283dbf38-tmp\") pod \"klusterlet-addon-workmgr-5d5b44c959-ckdh8\" (UID: \"d7170f4c-193a-4693-babf-1942283dbf38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:41:17.864891 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.864787 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qxtt\" (UniqueName: \"kubernetes.io/projected/d7170f4c-193a-4693-babf-1942283dbf38-kube-api-access-6qxtt\") pod \"klusterlet-addon-workmgr-5d5b44c959-ckdh8\" (UID: \"d7170f4c-193a-4693-babf-1942283dbf38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:41:17.965768 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.965739 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d7170f4c-193a-4693-babf-1942283dbf38-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d5b44c959-ckdh8\" (UID: \"d7170f4c-193a-4693-babf-1942283dbf38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:41:17.965880 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.965774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7170f4c-193a-4693-babf-1942283dbf38-tmp\") pod \"klusterlet-addon-workmgr-5d5b44c959-ckdh8\" (UID: \"d7170f4c-193a-4693-babf-1942283dbf38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:41:17.965880 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.965794 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qxtt\" (UniqueName: \"kubernetes.io/projected/d7170f4c-193a-4693-babf-1942283dbf38-kube-api-access-6qxtt\") pod \"klusterlet-addon-workmgr-5d5b44c959-ckdh8\" (UID: \"d7170f4c-193a-4693-babf-1942283dbf38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:41:17.966127 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.966108 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7170f4c-193a-4693-babf-1942283dbf38-tmp\") pod \"klusterlet-addon-workmgr-5d5b44c959-ckdh8\" (UID: \"d7170f4c-193a-4693-babf-1942283dbf38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:41:17.968150 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.968126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d7170f4c-193a-4693-babf-1942283dbf38-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d5b44c959-ckdh8\" (UID: \"d7170f4c-193a-4693-babf-1942283dbf38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:41:17.972893 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:17.972868 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qxtt\" (UniqueName: \"kubernetes.io/projected/d7170f4c-193a-4693-babf-1942283dbf38-kube-api-access-6qxtt\") pod \"klusterlet-addon-workmgr-5d5b44c959-ckdh8\" (UID: \"d7170f4c-193a-4693-babf-1942283dbf38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:41:18.050941 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:18.050876 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:41:18.165742 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:18.165697 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8"] Jul 22 18:41:18.169378 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:41:18.169349 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7170f4c_193a_4693_babf_1942283dbf38.slice/crio-81092625a8d8565d15f67433c9de1404fe6fb2ef4d18def9c7515949610cb039 WatchSource:0}: Error finding container 81092625a8d8565d15f67433c9de1404fe6fb2ef4d18def9c7515949610cb039: Status 404 returned error can't find the container with id 81092625a8d8565d15f67433c9de1404fe6fb2ef4d18def9c7515949610cb039 Jul 22 18:41:18.320410 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:18.320343 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" event={"ID":"d7170f4c-193a-4693-babf-1942283dbf38","Type":"ContainerStarted","Data":"81092625a8d8565d15f67433c9de1404fe6fb2ef4d18def9c7515949610cb039"} Jul 22 18:41:23.335066 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:23.335024 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" event={"ID":"d7170f4c-193a-4693-babf-1942283dbf38","Type":"ContainerStarted","Data":"1f898ef919dfb21a55b3e5b18d9080f6e4a78a8055b94cf300ee134365de2aa6"} Jul 22 18:41:23.335564 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:23.335221 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:41:23.336877 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:23.336857 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:41:23.352256 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:23.352219 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" podStartSLOduration=2.117946685 podStartE2EDuration="6.352208509s" totalImagesPullingTime="4.234261824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:41:23.352039517 +0000 UTC m=+93.799434749" watchObservedRunningTime="2026-07-22 18:41:23.352208509 +0000 UTC m=+93.799603741" Jul 22 18:41:27.654139 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:27.654105 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:41:27.709690 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:27.709668 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:41:29.750660 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:29.750625 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:41:29.750660 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:29.750664 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:41:29.751117 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:29.750721 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:41:29.751117 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:41:29.750784 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:41:29.751117 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:41:29.750806 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cd48d98-2wkxk: secret "image-registry-tls" not found Jul 22 18:41:29.751117 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:41:29.750807 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:41:29.751117 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:41:29.750821 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 18:41:29.751117 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:41:29.750861 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls podName:3f689b8a-ab35-4f36-a20d-047e5614eac1 nodeName:}" failed. No retries permitted until 2026-07-22 18:42:33.750845764 +0000 UTC m=+164.198240977 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls") pod "image-registry-97cd48d98-2wkxk" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1") : secret "image-registry-tls" not found Jul 22 18:41:29.751117 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:41:29.750875 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls podName:42cc3535-6546-4f2d-a502-9f3a97a63eaa nodeName:}" failed. No retries permitted until 2026-07-22 18:42:33.750869196 +0000 UTC m=+164.198264406 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls") pod "dns-default-x262w" (UID: "42cc3535-6546-4f2d-a502-9f3a97a63eaa") : secret "dns-default-metrics-tls" not found Jul 22 18:41:29.751117 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:41:29.750884 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert podName:b117ac7f-9bc8-46e2-a049-fdc145a52523 nodeName:}" failed. No retries permitted until 2026-07-22 18:42:33.750879192 +0000 UTC m=+164.198274403 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-78xp8" (UID: "b117ac7f-9bc8-46e2-a049-fdc145a52523") : secret "networking-console-plugin-cert" not found Jul 22 18:41:31.282198 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:31.282165 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cvmqq" Jul 22 18:41:59.746737 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:41:59.746675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:41:59.747263 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:41:59.746828 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 18:41:59.747263 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:41:59.746896 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs podName:d27e81e3-a9d6-4c4b-b138-3a4394204eb3 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:01.746879778 +0000 UTC m=+252.194274989 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs") pod "network-metrics-daemon-lc7qt" (UID: "d27e81e3-a9d6-4c4b-b138-3a4394204eb3") : secret "metrics-daemon-secret" not found Jul 22 18:42:28.950182 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:42:28.950138 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" podUID="3f689b8a-ab35-4f36-a20d-047e5614eac1" Jul 22 18:42:28.973426 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:42:28.973389 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" podUID="b117ac7f-9bc8-46e2-a049-fdc145a52523" Jul 22 18:42:28.979589 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:42:28.979558 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-x262w" podUID="42cc3535-6546-4f2d-a502-9f3a97a63eaa" Jul 22 18:42:29.489159 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:42:29.489129 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:42:29.489331 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:42:29.489136 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:42:29.489331 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:42:29.489136 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x262w" Jul 22 18:42:30.021465 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:42:30.021427 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:42:33.788177 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:42:33.788143 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:42:33.788177 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:42:33.788177 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:42:33.788736 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:42:33.788199 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:42:33.788736 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:42:33.788298 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:42:33.788736 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:42:33.788304 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:42:33.788736 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:42:33.788359 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 18:42:33.788736 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:42:33.788391 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls podName:42cc3535-6546-4f2d-a502-9f3a97a63eaa nodeName:}" failed. No retries permitted until 2026-07-22 18:44:35.788370613 +0000 UTC m=+286.235765841 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls") pod "dns-default-x262w" (UID: "42cc3535-6546-4f2d-a502-9f3a97a63eaa") : secret "dns-default-metrics-tls" not found Jul 22 18:42:33.788736 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:42:33.788316 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cd48d98-2wkxk: secret "image-registry-tls" not found Jul 22 18:42:33.788736 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:42:33.788411 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert podName:b117ac7f-9bc8-46e2-a049-fdc145a52523 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:35.788401736 +0000 UTC m=+286.235796946 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-78xp8" (UID: "b117ac7f-9bc8-46e2-a049-fdc145a52523") : secret "networking-console-plugin-cert" not found Jul 22 18:42:33.788736 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:42:33.788445 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls podName:3f689b8a-ab35-4f36-a20d-047e5614eac1 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:35.788431971 +0000 UTC m=+286.235827181 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls") pod "image-registry-97cd48d98-2wkxk" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1") : secret "image-registry-tls" not found Jul 22 18:42:42.000049 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:42:42.000012 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:43:22.609145 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:43:22.609119 2574 generic.go:356] "Generic (PLEG): container finished" podID="d7170f4c-193a-4693-babf-1942283dbf38" containerID="1f898ef919dfb21a55b3e5b18d9080f6e4a78a8055b94cf300ee134365de2aa6" exitCode=1 Jul 22 18:43:22.609466 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:43:22.609153 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" event={"ID":"d7170f4c-193a-4693-babf-1942283dbf38","Type":"ContainerDied","Data":"1f898ef919dfb21a55b3e5b18d9080f6e4a78a8055b94cf300ee134365de2aa6"} Jul 22 18:43:22.609466 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:43:22.609405 2574 scope.go:117] "RemoveContainer" containerID="1f898ef919dfb21a55b3e5b18d9080f6e4a78a8055b94cf300ee134365de2aa6" Jul 22 18:43:23.336051 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:43:23.336020 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:43:23.612858 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:43:23.612782 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" event={"ID":"d7170f4c-193a-4693-babf-1942283dbf38","Type":"ContainerStarted","Data":"ee25a4afb1ce7dc10ac12b3a9161543d63515a8695401b115836ac8444b867c7"} Jul 22 18:43:23.613206 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:43:23.613019 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:43:23.613580 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:43:23.613561 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d5b44c959-ckdh8" Jul 22 18:44:01.776436 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:01.776385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:44:01.777039 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:01.776553 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 18:44:01.777039 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:01.776647 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs podName:d27e81e3-a9d6-4c4b-b138-3a4394204eb3 nodeName:}" failed. No retries permitted until 2026-07-22 18:46:03.776625648 +0000 UTC m=+374.224020858 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs") pod "network-metrics-daemon-lc7qt" (UID: "d27e81e3-a9d6-4c4b-b138-3a4394204eb3") : secret "metrics-daemon-secret" not found Jul 22 18:44:19.730434 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.730351 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-kln74"] Jul 22 18:44:19.733358 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.733338 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-kln74" Jul 22 18:44:19.735481 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.735455 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xqqf2\"" Jul 22 18:44:19.741010 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.740989 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv"] Jul 22 18:44:19.743711 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.743694 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-kln74"] Jul 22 18:44:19.743791 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.743770 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" Jul 22 18:44:19.746905 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.746883 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 22 18:44:19.747431 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.747410 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h62ct\"" Jul 22 18:44:19.747431 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.747417 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 22 18:44:19.747594 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.747438 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 22 18:44:19.755076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.755055 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv"] Jul 22 18:44:19.802365 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.802340 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85bhl\" (UniqueName: \"kubernetes.io/projected/226afb1e-2876-4c1e-bd62-6a13c7f78c00-kube-api-access-85bhl\") pod \"cluster-samples-operator-79b7c55f87-bwcjv\" (UID: \"226afb1e-2876-4c1e-bd62-6a13c7f78c00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" Jul 22 18:44:19.802447 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.802432 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt9zw\" (UniqueName: \"kubernetes.io/projected/23267c19-b2ba-445c-ab01-20020cc6c7eb-kube-api-access-wt9zw\") pod \"network-check-source-7bc5b6bf85-kln74\" (UID: \"23267c19-b2ba-445c-ab01-20020cc6c7eb\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-kln74" Jul 22 18:44:19.802489 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.802466 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bwcjv\" (UID: \"226afb1e-2876-4c1e-bd62-6a13c7f78c00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" Jul 22 18:44:19.902848 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.902831 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85bhl\" (UniqueName: \"kubernetes.io/projected/226afb1e-2876-4c1e-bd62-6a13c7f78c00-kube-api-access-85bhl\") pod \"cluster-samples-operator-79b7c55f87-bwcjv\" (UID: \"226afb1e-2876-4c1e-bd62-6a13c7f78c00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" Jul 22 18:44:19.902922 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.902894 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt9zw\" (UniqueName: \"kubernetes.io/projected/23267c19-b2ba-445c-ab01-20020cc6c7eb-kube-api-access-wt9zw\") pod \"network-check-source-7bc5b6bf85-kln74\" (UID: \"23267c19-b2ba-445c-ab01-20020cc6c7eb\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-kln74" Jul 22 18:44:19.902922 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.902918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bwcjv\" (UID: \"226afb1e-2876-4c1e-bd62-6a13c7f78c00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" Jul 22 18:44:19.903025 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:19.903014 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 18:44:19.903072 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:19.903064 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls podName:226afb1e-2876-4c1e-bd62-6a13c7f78c00 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:20.403052207 +0000 UTC m=+270.850447417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bwcjv" (UID: "226afb1e-2876-4c1e-bd62-6a13c7f78c00") : secret "samples-operator-tls" not found Jul 22 18:44:19.912304 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.912283 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:19.913793 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.913776 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85bhl\" (UniqueName: \"kubernetes.io/projected/226afb1e-2876-4c1e-bd62-6a13c7f78c00-kube-api-access-85bhl\") pod \"cluster-samples-operator-79b7c55f87-bwcjv\" (UID: \"226afb1e-2876-4c1e-bd62-6a13c7f78c00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" Jul 22 18:44:19.913988 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.913954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt9zw\" (UniqueName: \"kubernetes.io/projected/23267c19-b2ba-445c-ab01-20020cc6c7eb-kube-api-access-wt9zw\") pod \"network-check-source-7bc5b6bf85-kln74\" (UID: \"23267c19-b2ba-445c-ab01-20020cc6c7eb\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-kln74" Jul 22 18:44:19.998586 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:19.998515 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:20.042134 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:20.042108 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-kln74" Jul 22 18:44:20.158582 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:20.158554 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-kln74"] Jul 22 18:44:20.161645 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:44:20.161570 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23267c19_b2ba_445c_ab01_20020cc6c7eb.slice/crio-e7a5684c0ab4079b86afb637900e116cb187396705584c73049bf5c6b0f7f5df WatchSource:0}: Error finding container e7a5684c0ab4079b86afb637900e116cb187396705584c73049bf5c6b0f7f5df: Status 404 returned error can't find the container with id e7a5684c0ab4079b86afb637900e116cb187396705584c73049bf5c6b0f7f5df Jul 22 18:44:20.406621 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:20.406537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bwcjv\" (UID: \"226afb1e-2876-4c1e-bd62-6a13c7f78c00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" Jul 22 18:44:20.406758 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:20.406678 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 18:44:20.406758 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:20.406738 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls podName:226afb1e-2876-4c1e-bd62-6a13c7f78c00 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:21.406719905 +0000 UTC m=+271.854115116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bwcjv" (UID: "226afb1e-2876-4c1e-bd62-6a13c7f78c00") : secret "samples-operator-tls" not found Jul 22 18:44:20.749764 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:20.749727 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-kln74" event={"ID":"23267c19-b2ba-445c-ab01-20020cc6c7eb","Type":"ContainerStarted","Data":"6fc45b77f9dcbe6da34f52bd9b1ce62a10ab6defeb50242fbe936048c78e0092"} Jul 22 18:44:20.750207 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:20.749799 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-kln74" event={"ID":"23267c19-b2ba-445c-ab01-20020cc6c7eb","Type":"ContainerStarted","Data":"e7a5684c0ab4079b86afb637900e116cb187396705584c73049bf5c6b0f7f5df"} Jul 22 18:44:20.766507 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:20.766459 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-kln74" podStartSLOduration=1.7664408969999998 podStartE2EDuration="1.766440897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:44:20.764359648 +0000 UTC m=+271.211754880" watchObservedRunningTime="2026-07-22 18:44:20.766440897 +0000 UTC m=+271.213836130" Jul 22 18:44:21.414874 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:21.414841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bwcjv\" (UID: \"226afb1e-2876-4c1e-bd62-6a13c7f78c00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" Jul 22 18:44:21.415073 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:21.415004 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 18:44:21.415124 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:21.415085 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls podName:226afb1e-2876-4c1e-bd62-6a13c7f78c00 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:23.415065869 +0000 UTC m=+273.862461078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bwcjv" (UID: "226afb1e-2876-4c1e-bd62-6a13c7f78c00") : secret "samples-operator-tls" not found Jul 22 18:44:22.895515 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:22.895483 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:22.946294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:22.946265 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:23.429213 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:23.429175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bwcjv\" (UID: \"226afb1e-2876-4c1e-bd62-6a13c7f78c00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" Jul 22 18:44:23.429391 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:23.429306 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 18:44:23.429391 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:23.429360 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls podName:226afb1e-2876-4c1e-bd62-6a13c7f78c00 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:27.429346326 +0000 UTC m=+277.876741536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bwcjv" (UID: "226afb1e-2876-4c1e-bd62-6a13c7f78c00") : secret "samples-operator-tls" not found Jul 22 18:44:25.211029 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.210993 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:25.231145 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.231117 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vhvgj"] Jul 22 18:44:25.233990 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.233961 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vhvgj" Jul 22 18:44:25.235949 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.235929 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 22 18:44:25.236792 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.236774 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kjbrm\"" Jul 22 18:44:25.236855 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.236829 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 22 18:44:25.242634 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.242612 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vhvgj"] Jul 22 18:44:25.280748 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.280726 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hgnhk"] Jul 22 18:44:25.283790 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.283775 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.285942 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.285917 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 18:44:25.286041 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.286017 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 18:44:25.286090 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.286065 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 18:44:25.286146 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.286129 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 18:44:25.286208 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.286192 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rvpd8\"" Jul 22 18:44:25.294846 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.294823 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hgnhk"] Jul 22 18:44:25.301424 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.301407 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:25.341850 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.341826 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c44ab18a-bae2-4048-a505-bb14abbe90b7-data-volume\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.341938 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.341853 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.341938 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.341904 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.342056 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.341948 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hc7c\" (UniqueName: \"kubernetes.io/projected/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-api-access-8hc7c\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.342056 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.342028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c44ab18a-bae2-4048-a505-bb14abbe90b7-crio-socket\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.342134 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.342083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7spxj\" (UniqueName: \"kubernetes.io/projected/16d275cd-541e-4cfa-80ab-68c617f381bc-kube-api-access-7spxj\") pod \"migrator-7cc9c7ffd7-vhvgj\" (UID: \"16d275cd-541e-4cfa-80ab-68c617f381bc\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vhvgj" Jul 22 18:44:25.392887 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.392860 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:25.443098 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.443061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c44ab18a-bae2-4048-a505-bb14abbe90b7-data-volume\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.443098 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.443103 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.443270 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.443143 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.443270 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.443160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hc7c\" (UniqueName: \"kubernetes.io/projected/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-api-access-8hc7c\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.443270 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.443184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c44ab18a-bae2-4048-a505-bb14abbe90b7-crio-socket\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.443270 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:25.443202 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 18:44:25.443270 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.443218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7spxj\" (UniqueName: \"kubernetes.io/projected/16d275cd-541e-4cfa-80ab-68c617f381bc-kube-api-access-7spxj\") pod \"migrator-7cc9c7ffd7-vhvgj\" (UID: \"16d275cd-541e-4cfa-80ab-68c617f381bc\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vhvgj" Jul 22 18:44:25.443492 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:25.443286 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm podName:c44ab18a-bae2-4048-a505-bb14abbe90b7 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:25.943266082 +0000 UTC m=+276.390661317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgnhk" (UID: "c44ab18a-bae2-4048-a505-bb14abbe90b7") : configmap "kube-rbac-proxy" not found Jul 22 18:44:25.443492 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.443310 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c44ab18a-bae2-4048-a505-bb14abbe90b7-crio-socket\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.443492 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:25.443336 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 18:44:25.443492 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:25.443383 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls podName:c44ab18a-bae2-4048-a505-bb14abbe90b7 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:25.943371436 +0000 UTC m=+276.390766647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgnhk" (UID: "c44ab18a-bae2-4048-a505-bb14abbe90b7") : secret "insights-runtime-extractor-tls" not found Jul 22 18:44:25.443492 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.443421 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c44ab18a-bae2-4048-a505-bb14abbe90b7-data-volume\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.453437 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.453413 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hc7c\" (UniqueName: \"kubernetes.io/projected/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-api-access-8hc7c\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.453437 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.453423 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7spxj\" (UniqueName: \"kubernetes.io/projected/16d275cd-541e-4cfa-80ab-68c617f381bc-kube-api-access-7spxj\") pod \"migrator-7cc9c7ffd7-vhvgj\" (UID: \"16d275cd-541e-4cfa-80ab-68c617f381bc\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vhvgj" Jul 22 18:44:25.543551 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.543518 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vhvgj" Jul 22 18:44:25.663580 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.663553 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vhvgj"] Jul 22 18:44:25.666557 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:44:25.666535 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d275cd_541e_4cfa_80ab_68c617f381bc.slice/crio-eeff67190cd248d4ff808946fed99bad2a32d99251906772da1033bab175deb5 WatchSource:0}: Error finding container eeff67190cd248d4ff808946fed99bad2a32d99251906772da1033bab175deb5: Status 404 returned error can't find the container with id eeff67190cd248d4ff808946fed99bad2a32d99251906772da1033bab175deb5 Jul 22 18:44:25.729022 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.728995 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:25.767387 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.767362 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vhvgj" event={"ID":"16d275cd-541e-4cfa-80ab-68c617f381bc","Type":"ContainerStarted","Data":"eeff67190cd248d4ff808946fed99bad2a32d99251906772da1033bab175deb5"} Jul 22 18:44:25.783729 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.783706 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:25.946985 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.946909 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.946985 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:25.946956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:25.947120 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:25.947031 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 18:44:25.947120 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:25.947092 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 18:44:25.947120 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:25.947095 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm podName:c44ab18a-bae2-4048-a505-bb14abbe90b7 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:26.947080699 +0000 UTC m=+277.394475909 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgnhk" (UID: "c44ab18a-bae2-4048-a505-bb14abbe90b7") : configmap "kube-rbac-proxy" not found Jul 22 18:44:25.947218 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:25.947136 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls podName:c44ab18a-bae2-4048-a505-bb14abbe90b7 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:26.947126717 +0000 UTC m=+277.394521928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgnhk" (UID: "c44ab18a-bae2-4048-a505-bb14abbe90b7") : secret "insights-runtime-extractor-tls" not found Jul 22 18:44:26.443941 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:26.443906 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:26.654247 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:26.654212 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:26.954861 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:26.954820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:26.955056 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:26.954899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:26.955056 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:26.954958 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 18:44:26.955056 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:26.955051 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm podName:c44ab18a-bae2-4048-a505-bb14abbe90b7 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:28.955030359 +0000 UTC m=+279.402425574 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgnhk" (UID: "c44ab18a-bae2-4048-a505-bb14abbe90b7") : configmap "kube-rbac-proxy" not found Jul 22 18:44:26.955235 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:26.955072 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 18:44:26.955235 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:26.955126 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls podName:c44ab18a-bae2-4048-a505-bb14abbe90b7 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:28.955114182 +0000 UTC m=+279.402509392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgnhk" (UID: "c44ab18a-bae2-4048-a505-bb14abbe90b7") : secret "insights-runtime-extractor-tls" not found Jul 22 18:44:27.458048 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:27.458008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bwcjv\" (UID: \"226afb1e-2876-4c1e-bd62-6a13c7f78c00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" Jul 22 18:44:27.458422 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:27.458125 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 18:44:27.458422 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:27.458177 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls podName:226afb1e-2876-4c1e-bd62-6a13c7f78c00 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:35.458162789 +0000 UTC m=+285.905558000 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bwcjv" (UID: "226afb1e-2876-4c1e-bd62-6a13c7f78c00") : secret "samples-operator-tls" not found Jul 22 18:44:27.775883 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:27.775851 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vhvgj" event={"ID":"16d275cd-541e-4cfa-80ab-68c617f381bc","Type":"ContainerStarted","Data":"f9018336123a414cbc432d2da3f0dc939000d2606be8d0b978f9635ee00e7fd4"} Jul 22 18:44:27.776071 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:27.775905 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vhvgj" event={"ID":"16d275cd-541e-4cfa-80ab-68c617f381bc","Type":"ContainerStarted","Data":"81faba49d11dfc1f8e6d460cc732386e1194ccaa634d042d9f303f79d000f191"} Jul 22 18:44:27.790338 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:27.790293 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vhvgj" podStartSLOduration=1.441811808 podStartE2EDuration="2.79027857s" totalImagesPullingTime="1.348466762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:44:27.789162689 +0000 UTC m=+278.236557920" watchObservedRunningTime="2026-07-22 18:44:27.79027857 +0000 UTC m=+278.237673802" Jul 22 18:44:27.904938 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:27.904909 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:27.964818 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:27.964797 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:28.031689 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.031629 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wx689"] Jul 22 18:44:28.034537 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.034520 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-wx689" Jul 22 18:44:28.036622 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.036601 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 22 18:44:28.036730 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.036655 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 22 18:44:28.036914 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.036896 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 22 18:44:28.037434 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.037419 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xg9jt\"" Jul 22 18:44:28.037513 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.037501 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 22 18:44:28.042174 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.042150 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wx689"] Jul 22 18:44:28.065909 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.065871 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:28.164506 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.164473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/20eacc93-bb03-43bd-9677-7427601e252d-signing-cabundle\") pod \"service-ca-58598695f7-wx689\" (UID: \"20eacc93-bb03-43bd-9677-7427601e252d\") " pod="openshift-service-ca/service-ca-58598695f7-wx689" Jul 22 18:44:28.164660 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.164516 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmv5m\" (UniqueName: \"kubernetes.io/projected/20eacc93-bb03-43bd-9677-7427601e252d-kube-api-access-tmv5m\") pod \"service-ca-58598695f7-wx689\" (UID: \"20eacc93-bb03-43bd-9677-7427601e252d\") " pod="openshift-service-ca/service-ca-58598695f7-wx689" Jul 22 18:44:28.164660 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.164649 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/20eacc93-bb03-43bd-9677-7427601e252d-signing-key\") pod \"service-ca-58598695f7-wx689\" (UID: \"20eacc93-bb03-43bd-9677-7427601e252d\") " pod="openshift-service-ca/service-ca-58598695f7-wx689" Jul 22 18:44:28.265132 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.265098 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/20eacc93-bb03-43bd-9677-7427601e252d-signing-key\") pod \"service-ca-58598695f7-wx689\" (UID: \"20eacc93-bb03-43bd-9677-7427601e252d\") " pod="openshift-service-ca/service-ca-58598695f7-wx689" Jul 22 18:44:28.265279 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.265163 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/20eacc93-bb03-43bd-9677-7427601e252d-signing-cabundle\") pod \"service-ca-58598695f7-wx689\" (UID: \"20eacc93-bb03-43bd-9677-7427601e252d\") " pod="openshift-service-ca/service-ca-58598695f7-wx689" Jul 22 18:44:28.265335 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.265316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmv5m\" (UniqueName: \"kubernetes.io/projected/20eacc93-bb03-43bd-9677-7427601e252d-kube-api-access-tmv5m\") pod \"service-ca-58598695f7-wx689\" (UID: \"20eacc93-bb03-43bd-9677-7427601e252d\") " pod="openshift-service-ca/service-ca-58598695f7-wx689" Jul 22 18:44:28.265749 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.265734 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/20eacc93-bb03-43bd-9677-7427601e252d-signing-cabundle\") pod \"service-ca-58598695f7-wx689\" (UID: \"20eacc93-bb03-43bd-9677-7427601e252d\") " pod="openshift-service-ca/service-ca-58598695f7-wx689" Jul 22 18:44:28.267407 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.267387 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/20eacc93-bb03-43bd-9677-7427601e252d-signing-key\") pod \"service-ca-58598695f7-wx689\" (UID: \"20eacc93-bb03-43bd-9677-7427601e252d\") " pod="openshift-service-ca/service-ca-58598695f7-wx689" Jul 22 18:44:28.272576 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.272555 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmv5m\" (UniqueName: \"kubernetes.io/projected/20eacc93-bb03-43bd-9677-7427601e252d-kube-api-access-tmv5m\") pod \"service-ca-58598695f7-wx689\" (UID: \"20eacc93-bb03-43bd-9677-7427601e252d\") " pod="openshift-service-ca/service-ca-58598695f7-wx689" Jul 22 18:44:28.295143 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.295093 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7kgx_daf45312-be61-4ecc-bf68-ee249f228c2b/dns-node-resolver/0.log" Jul 22 18:44:28.343362 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.343332 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-wx689" Jul 22 18:44:28.462860 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.462834 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wx689"] Jul 22 18:44:28.465097 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:44:28.465072 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20eacc93_bb03_43bd_9677_7427601e252d.slice/crio-3d13cbe3aa3ade44fe7def1800f64ed469ad28d50d970078d323f938e4a9a146 WatchSource:0}: Error finding container 3d13cbe3aa3ade44fe7def1800f64ed469ad28d50d970078d323f938e4a9a146: Status 404 returned error can't find the container with id 3d13cbe3aa3ade44fe7def1800f64ed469ad28d50d970078d323f938e4a9a146 Jul 22 18:44:28.780035 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.780006 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-wx689" event={"ID":"20eacc93-bb03-43bd-9677-7427601e252d","Type":"ContainerStarted","Data":"3d13cbe3aa3ade44fe7def1800f64ed469ad28d50d970078d323f938e4a9a146"} Jul 22 18:44:28.970717 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.970675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:28.970896 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:28.970772 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 18:44:28.970896 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:28.970852 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 18:44:28.970896 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:28.970757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:28.970896 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:28.970879 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm podName:c44ab18a-bae2-4048-a505-bb14abbe90b7 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:32.970859824 +0000 UTC m=+283.418255055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgnhk" (UID: "c44ab18a-bae2-4048-a505-bb14abbe90b7") : configmap "kube-rbac-proxy" not found Jul 22 18:44:28.971093 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:28.970984 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls podName:c44ab18a-bae2-4048-a505-bb14abbe90b7 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:32.970956268 +0000 UTC m=+283.418351486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgnhk" (UID: "c44ab18a-bae2-4048-a505-bb14abbe90b7") : secret "insights-runtime-extractor-tls" not found Jul 22 18:44:29.088671 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:29.088588 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:29.284626 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:29.284595 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gg7vx_4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4/node-ca/0.log" Jul 22 18:44:29.683882 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:29.683853 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8ph6g_7fe1adde-5816-463f-aa6a-7769d7569b7f/serve-healthcheck-canary/0.log" Jul 22 18:44:30.284066 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:30.284036 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-vhvgj_16d275cd-541e-4cfa-80ab-68c617f381bc/migrator/0.log" Jul 22 18:44:30.484710 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:30.484677 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-vhvgj_16d275cd-541e-4cfa-80ab-68c617f381bc/graceful-termination/0.log" Jul 22 18:44:30.787101 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:30.787069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-wx689" event={"ID":"20eacc93-bb03-43bd-9677-7427601e252d","Type":"ContainerStarted","Data":"dfd7d952771d1a9daebcdd28841805897edde489520735fbc468174181d33994"} Jul 22 18:44:30.801415 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:30.801359 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-wx689" podStartSLOduration=0.9819904 podStartE2EDuration="2.801339388s" totalImagesPullingTime="1.819348988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:44:30.799846881 +0000 UTC m=+281.247242124" watchObservedRunningTime="2026-07-22 18:44:30.801339388 +0000 UTC m=+281.248734620" Jul 22 18:44:31.655526 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:31.655471 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:31.755935 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:31.755899 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:31.888839 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:31.888798 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:32.022723 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:32.022679 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:32.081499 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:32.081468 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:32.160594 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:32.160568 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:32.419505 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:32.419425 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:32.489521 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:32.489488 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" podUID="b117ac7f-9bc8-46e2-a049-fdc145a52523" Jul 22 18:44:32.489666 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:32.489643 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-x262w" podUID="42cc3535-6546-4f2d-a502-9f3a97a63eaa" Jul 22 18:44:32.489723 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:32.489702 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" podUID="3f689b8a-ab35-4f36-a20d-047e5614eac1" Jul 22 18:44:32.793923 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:32.793894 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:44:32.794081 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:32.793894 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:44:32.794081 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:32.793895 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x262w" Jul 22 18:44:33.003379 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:33.003349 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:33.003790 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:33.003409 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:33.003790 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:33.003465 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 18:44:33.003790 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:33.003526 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 18:44:33.003790 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:33.003548 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm podName:c44ab18a-bae2-4048-a505-bb14abbe90b7 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:41.003529772 +0000 UTC m=+291.450924995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgnhk" (UID: "c44ab18a-bae2-4048-a505-bb14abbe90b7") : configmap "kube-rbac-proxy" not found Jul 22 18:44:33.003790 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:33.003570 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls podName:c44ab18a-bae2-4048-a505-bb14abbe90b7 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:41.00355986 +0000 UTC m=+291.450955070 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgnhk" (UID: "c44ab18a-bae2-4048-a505-bb14abbe90b7") : secret "insights-runtime-extractor-tls" not found Jul 22 18:44:33.421166 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:33.421131 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:35.117822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.117769 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:35.178196 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.178168 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:35.241931 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.241904 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:35.415238 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.415164 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:35.522394 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.522348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bwcjv\" (UID: \"226afb1e-2876-4c1e-bd62-6a13c7f78c00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" Jul 22 18:44:35.524754 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.524724 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226afb1e-2876-4c1e-bd62-6a13c7f78c00-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bwcjv\" (UID: \"226afb1e-2876-4c1e-bd62-6a13c7f78c00\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" Jul 22 18:44:35.610889 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.610855 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:35.652011 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.651987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" Jul 22 18:44:35.775006 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.774908 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv"] Jul 22 18:44:35.824459 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.824434 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:44:35.824550 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.824510 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:44:35.824550 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.824531 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:44:35.824668 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:35.824654 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 18:44:35.824735 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:35.824726 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert podName:b117ac7f-9bc8-46e2-a049-fdc145a52523 nodeName:}" failed. No retries permitted until 2026-07-22 18:46:37.82471283 +0000 UTC m=+408.272108041 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-78xp8" (UID: "b117ac7f-9bc8-46e2-a049-fdc145a52523") : secret "networking-console-plugin-cert" not found Jul 22 18:44:35.826572 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.826543 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cc3535-6546-4f2d-a502-9f3a97a63eaa-metrics-tls\") pod \"dns-default-x262w\" (UID: \"42cc3535-6546-4f2d-a502-9f3a97a63eaa\") " pod="openshift-dns/dns-default-x262w" Jul 22 18:44:35.826701 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.826686 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") pod \"image-registry-97cd48d98-2wkxk\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:44:35.842851 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.842826 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:35.946213 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:35.946182 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:36.013892 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.013861 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:36.096560 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.096531 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hbkg4\"" Jul 22 18:44:36.096704 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.096542 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bzh2s\"" Jul 22 18:44:36.105233 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.105202 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:44:36.105233 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.105220 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x262w" Jul 22 18:44:36.234599 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.234570 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-97cd48d98-2wkxk"] Jul 22 18:44:36.237929 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:44:36.237903 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f689b8a_ab35_4f36_a20d_047e5614eac1.slice/crio-6029aefaaa59dbee6ba89d586267395107d479bcef254666741648bbc315ffcf WatchSource:0}: Error finding container 6029aefaaa59dbee6ba89d586267395107d479bcef254666741648bbc315ffcf: Status 404 returned error can't find the container with id 6029aefaaa59dbee6ba89d586267395107d479bcef254666741648bbc315ffcf Jul 22 18:44:36.249018 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.248997 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x262w"] Jul 22 18:44:36.251213 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:44:36.251186 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42cc3535_6546_4f2d_a502_9f3a97a63eaa.slice/crio-65d8f840c1f9b3eca94c70a7c5c7d74779afe882ae74001d0c89c411bdec635a WatchSource:0}: Error finding container 65d8f840c1f9b3eca94c70a7c5c7d74779afe882ae74001d0c89c411bdec635a: Status 404 returned error can't find the container with id 65d8f840c1f9b3eca94c70a7c5c7d74779afe882ae74001d0c89c411bdec635a Jul 22 18:44:36.414830 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.414781 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:36.811342 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.811302 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" event={"ID":"3f689b8a-ab35-4f36-a20d-047e5614eac1","Type":"ContainerStarted","Data":"8daa6750e537e39d1bcc84537e115a669d3c6120822f3a6b1b0375e8f85710a8"} Jul 22 18:44:36.811529 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.811362 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" event={"ID":"3f689b8a-ab35-4f36-a20d-047e5614eac1","Type":"ContainerStarted","Data":"6029aefaaa59dbee6ba89d586267395107d479bcef254666741648bbc315ffcf"} Jul 22 18:44:36.811529 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.811388 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:44:36.812866 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.812838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" event={"ID":"226afb1e-2876-4c1e-bd62-6a13c7f78c00","Type":"ContainerStarted","Data":"e357c8f215cff496f2710bf7473d912aa05f0b7fd11f27431086644f157eb20b"} Jul 22 18:44:36.814025 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.813998 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x262w" event={"ID":"42cc3535-6546-4f2d-a502-9f3a97a63eaa","Type":"ContainerStarted","Data":"65d8f840c1f9b3eca94c70a7c5c7d74779afe882ae74001d0c89c411bdec635a"} Jul 22 18:44:36.821353 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.821327 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:36.832568 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:36.832526 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" podStartSLOduration=286.832512585 podStartE2EDuration="4m46.832512585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:44:36.831598131 +0000 UTC m=+287.278993366" watchObservedRunningTime="2026-07-22 18:44:36.832512585 +0000 UTC m=+287.279907817" Jul 22 18:44:37.216869 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:37.216794 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:37.425152 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:37.425119 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:37.819963 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:37.819929 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" event={"ID":"226afb1e-2876-4c1e-bd62-6a13c7f78c00","Type":"ContainerStarted","Data":"4308761204b2b2a1080ac56e42991f9e0c7bd7437c9ec8fc8bcbfa57a5e116f8"} Jul 22 18:44:37.820190 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:37.820011 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" event={"ID":"226afb1e-2876-4c1e-bd62-6a13c7f78c00","Type":"ContainerStarted","Data":"79ed377ec42d30aab7cb2285be87238c3a399b4915c95c67281f311ec6330022"} Jul 22 18:44:37.850858 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:37.850810 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bwcjv" podStartSLOduration=17.013833618 podStartE2EDuration="18.85079533s" totalImagesPullingTime="1.836961712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:44:37.849344375 +0000 UTC m=+288.296739608" watchObservedRunningTime="2026-07-22 18:44:37.85079533 +0000 UTC m=+288.298190593" Jul 22 18:44:37.859621 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:37.859599 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:38.268193 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:38.268165 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:38.437601 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:38.437570 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:38.496763 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:38.496730 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:39.014041 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:39.014008 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:39.219627 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:39.219580 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:39.420207 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:39.420134 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:39.618924 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:39.618866 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:39.828423 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:39.828393 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x262w" event={"ID":"42cc3535-6546-4f2d-a502-9f3a97a63eaa","Type":"ContainerStarted","Data":"0329cc20fdbaef0f0da058213d5d3957d69bfb06d3d3d092eb89c6aad2694233"} Jul 22 18:44:39.828566 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:39.828441 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x262w" event={"ID":"42cc3535-6546-4f2d-a502-9f3a97a63eaa","Type":"ContainerStarted","Data":"e92ebcce8eabb121526d468d4909c3e2968f8aced72b859df784f9c76d20bd6c"} Jul 22 18:44:39.828566 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:39.828533 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x262w" Jul 22 18:44:39.844607 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:39.844554 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x262w" podStartSLOduration=251.66941076 podStartE2EDuration="4m14.844543261s" totalImagesPullingTime="3.175132501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:44:39.843832488 +0000 UTC m=+290.291227722" watchObservedRunningTime="2026-07-22 18:44:39.844543261 +0000 UTC m=+290.291938471" Jul 22 18:44:40.817126 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:40.817095 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:41.013018 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:41.012987 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:41.068183 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:41.068113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:41.068183 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:41.068170 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:41.068330 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:41.068286 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 18:44:41.068369 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:41.068349 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls podName:c44ab18a-bae2-4048-a505-bb14abbe90b7 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:57.068332165 +0000 UTC m=+307.515727384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgnhk" (UID: "c44ab18a-bae2-4048-a505-bb14abbe90b7") : secret "insights-runtime-extractor-tls" not found Jul 22 18:44:41.068592 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:41.068573 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c44ab18a-bae2-4048-a505-bb14abbe90b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:41.213918 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:41.213887 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:41.411145 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:41.411064 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:41.610705 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:41.610669 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:42.215153 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:42.215123 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:42.413375 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:42.413349 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:45.001294 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:45.001259 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-lc7qt" podUID="d27e81e3-a9d6-4c4b-b138-3a4394204eb3" Jul 22 18:44:45.596032 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:45.595992 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:45.703671 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:45.703642 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:45.906584 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:45.906509 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:46.110417 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:46.110392 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:46.165647 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:46.165590 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:46.483615 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:46.483590 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:46.536583 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:46.536557 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:46.700631 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:46.700602 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:46.803099 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:46.803032 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:46.902485 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:46.902456 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:46.998105 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:46.998078 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:47.298515 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:47.298480 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:47.494868 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:47.494836 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:47.730084 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:47.730053 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:48.106738 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:48.106671 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:48.211569 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:48.211541 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:48.351604 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:48.351576 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:48.407167 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:48.407105 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:49.834655 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:49.834625 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x262w" Jul 22 18:44:49.884177 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:49.884149 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 18:44:54.734497 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:54.734456 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:55.747580 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:55.747546 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc"] Jul 22 18:44:55.749755 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:55.749738 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc" Jul 22 18:44:55.751965 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:55.751923 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 22 18:44:55.752123 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:55.751998 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fpxhq\"" Jul 22 18:44:55.759806 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:55.759766 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc"] Jul 22 18:44:55.874768 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:55.874736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/87e51017-f0f2-43c4-ab04-9f98e893b8e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-sznbc\" (UID: \"87e51017-f0f2-43c4-ab04-9f98e893b8e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc" Jul 22 18:44:55.975841 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:55.975813 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/87e51017-f0f2-43c4-ab04-9f98e893b8e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-sznbc\" (UID: \"87e51017-f0f2-43c4-ab04-9f98e893b8e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc" Jul 22 18:44:55.976008 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:55.975955 2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 22 18:44:55.976068 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:44:55.976028 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87e51017-f0f2-43c4-ab04-9f98e893b8e8-tls-certificates podName:87e51017-f0f2-43c4-ab04-9f98e893b8e8 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:56.476012541 +0000 UTC m=+306.923407751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/87e51017-f0f2-43c4-ab04-9f98e893b8e8-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-sznbc" (UID: "87e51017-f0f2-43c4-ab04-9f98e893b8e8") : secret "prometheus-operator-admission-webhook-tls" not found Jul 22 18:44:56.110273 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:56.110201 2574 patch_prober.go:28] interesting pod/image-registry-97cd48d98-2wkxk container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 22 18:44:56.110273 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:56.110259 2574 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" podUID="3f689b8a-ab35-4f36-a20d-047e5614eac1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 18:44:56.480814 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:56.480777 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/87e51017-f0f2-43c4-ab04-9f98e893b8e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-sznbc\" (UID: \"87e51017-f0f2-43c4-ab04-9f98e893b8e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc" Jul 22 18:44:56.483119 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:56.483101 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/87e51017-f0f2-43c4-ab04-9f98e893b8e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-sznbc\" (UID: \"87e51017-f0f2-43c4-ab04-9f98e893b8e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc" Jul 22 18:44:56.658643 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:56.658605 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc" Jul 22 18:44:56.770683 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:56.770654 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:56.776481 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:56.776451 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc"] Jul 22 18:44:56.780677 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:44:56.780648 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e51017_f0f2_43c4_ab04_9f98e893b8e8.slice/crio-671d754f370f7cc9be039bdc117d641ed1e1042bcaa670eb0c1e51d903eb0607 WatchSource:0}: Error finding container 671d754f370f7cc9be039bdc117d641ed1e1042bcaa670eb0c1e51d903eb0607: Status 404 returned error can't find the container with id 671d754f370f7cc9be039bdc117d641ed1e1042bcaa670eb0c1e51d903eb0607 Jul 22 18:44:56.782564 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:56.782549 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 18:44:56.879460 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:56.879424 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc" event={"ID":"87e51017-f0f2-43c4-ab04-9f98e893b8e8","Type":"ContainerStarted","Data":"671d754f370f7cc9be039bdc117d641ed1e1042bcaa670eb0c1e51d903eb0607"} Jul 22 18:44:57.085724 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:57.085643 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:57.087884 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:57.087856 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c44ab18a-bae2-4048-a505-bb14abbe90b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgnhk\" (UID: \"c44ab18a-bae2-4048-a505-bb14abbe90b7\") " pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:57.093757 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:57.093738 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rvpd8\"" Jul 22 18:44:57.102432 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:57.102413 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hgnhk" Jul 22 18:44:57.123541 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:57.123513 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:57.181893 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:57.181859 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:57.221893 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:57.221870 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hgnhk"] Jul 22 18:44:57.224643 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:44:57.224610 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44ab18a_bae2_4048_a505_bb14abbe90b7.slice/crio-b316a0756524704b5fee7f4babdfbcc3f3019ff2efca39932c24daabc700dbb1 WatchSource:0}: Error finding container b316a0756524704b5fee7f4babdfbcc3f3019ff2efca39932c24daabc700dbb1: Status 404 returned error can't find the container with id b316a0756524704b5fee7f4babdfbcc3f3019ff2efca39932c24daabc700dbb1 Jul 22 18:44:57.825552 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:57.825527 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:44:57.884769 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:57.884735 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgnhk" event={"ID":"c44ab18a-bae2-4048-a505-bb14abbe90b7","Type":"ContainerStarted","Data":"b316a0756524704b5fee7f4babdfbcc3f3019ff2efca39932c24daabc700dbb1"} Jul 22 18:44:58.076656 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:58.076586 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:58.176683 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:58.176643 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:58.891199 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:58.891146 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc" event={"ID":"87e51017-f0f2-43c4-ab04-9f98e893b8e8","Type":"ContainerStarted","Data":"a75a063dcb0d8e08b2556f39716ad9e4b1d9d83c6433acba1c4092d84308e162"} Jul 22 18:44:58.891670 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:58.891557 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc" Jul 22 18:44:58.897472 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:58.897414 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc" Jul 22 18:44:58.906577 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:58.906526 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sznbc" podStartSLOduration=2.572436808 podStartE2EDuration="3.906507549s" totalImagesPullingTime="1.334070741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:44:58.904513722 +0000 UTC m=+309.351908955" watchObservedRunningTime="2026-07-22 18:44:58.906507549 +0000 UTC m=+309.353902783" Jul 22 18:44:58.983623 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:58.983589 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:58.999648 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:58.999622 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:44:59.047129 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.047105 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:44:59.804622 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.804583 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6"] Jul 22 18:44:59.807726 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.807707 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:44:59.809814 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.809788 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 22 18:44:59.809908 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.809836 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 18:44:59.810651 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.810627 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qchgw\"" Jul 22 18:44:59.810754 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.810640 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 18:44:59.810754 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.810687 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 18:44:59.810754 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.810691 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 22 18:44:59.817587 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.817564 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6"] Jul 22 18:44:59.910347 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.910311 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/890548e6-01a9-4d90-a6b3-781b15f8ec79-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-hxjw6\" (UID: \"890548e6-01a9-4d90-a6b3-781b15f8ec79\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:44:59.910721 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.910365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjmrj\" (UniqueName: \"kubernetes.io/projected/890548e6-01a9-4d90-a6b3-781b15f8ec79-kube-api-access-fjmrj\") pod \"prometheus-operator-5945b5cfd-hxjw6\" (UID: \"890548e6-01a9-4d90-a6b3-781b15f8ec79\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:44:59.910721 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.910465 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/890548e6-01a9-4d90-a6b3-781b15f8ec79-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-hxjw6\" (UID: \"890548e6-01a9-4d90-a6b3-781b15f8ec79\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:44:59.910721 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:44:59.910492 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/890548e6-01a9-4d90-a6b3-781b15f8ec79-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-hxjw6\" (UID: \"890548e6-01a9-4d90-a6b3-781b15f8ec79\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:45:00.011099 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:00.011063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/890548e6-01a9-4d90-a6b3-781b15f8ec79-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-hxjw6\" (UID: \"890548e6-01a9-4d90-a6b3-781b15f8ec79\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:45:00.011279 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:00.011112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/890548e6-01a9-4d90-a6b3-781b15f8ec79-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-hxjw6\" (UID: \"890548e6-01a9-4d90-a6b3-781b15f8ec79\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:45:00.011279 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:00.011152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/890548e6-01a9-4d90-a6b3-781b15f8ec79-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-hxjw6\" (UID: \"890548e6-01a9-4d90-a6b3-781b15f8ec79\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:45:00.011279 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:00.011192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjmrj\" (UniqueName: \"kubernetes.io/projected/890548e6-01a9-4d90-a6b3-781b15f8ec79-kube-api-access-fjmrj\") pod \"prometheus-operator-5945b5cfd-hxjw6\" (UID: \"890548e6-01a9-4d90-a6b3-781b15f8ec79\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:45:00.011420 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:45:00.011290 2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 22 18:45:00.011420 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:45:00.011361 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/890548e6-01a9-4d90-a6b3-781b15f8ec79-prometheus-operator-tls podName:890548e6-01a9-4d90-a6b3-781b15f8ec79 nodeName:}" failed. No retries permitted until 2026-07-22 18:45:00.511339968 +0000 UTC m=+310.958735193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/890548e6-01a9-4d90-a6b3-781b15f8ec79-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-hxjw6" (UID: "890548e6-01a9-4d90-a6b3-781b15f8ec79") : secret "prometheus-operator-tls" not found Jul 22 18:45:00.012078 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:00.012054 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/890548e6-01a9-4d90-a6b3-781b15f8ec79-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-hxjw6\" (UID: \"890548e6-01a9-4d90-a6b3-781b15f8ec79\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:45:00.013697 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:00.013672 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/890548e6-01a9-4d90-a6b3-781b15f8ec79-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-hxjw6\" (UID: \"890548e6-01a9-4d90-a6b3-781b15f8ec79\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:45:00.021983 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:00.021947 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjmrj\" (UniqueName: \"kubernetes.io/projected/890548e6-01a9-4d90-a6b3-781b15f8ec79-kube-api-access-fjmrj\") pod \"prometheus-operator-5945b5cfd-hxjw6\" (UID: \"890548e6-01a9-4d90-a6b3-781b15f8ec79\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:45:00.078556 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:00.078488 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:00.140023 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:00.139952 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:00.436384 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:00.436281 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:00.516542 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:00.516502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/890548e6-01a9-4d90-a6b3-781b15f8ec79-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-hxjw6\" (UID: \"890548e6-01a9-4d90-a6b3-781b15f8ec79\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:45:00.519195 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:00.519173 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/890548e6-01a9-4d90-a6b3-781b15f8ec79-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-hxjw6\" (UID: \"890548e6-01a9-4d90-a6b3-781b15f8ec79\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:45:00.717346 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:00.717266 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" Jul 22 18:45:01.025572 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:01.025540 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:01.080914 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:01.080859 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:02.587817 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:02.587795 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6"] Jul 22 18:45:02.590565 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:45:02.590541 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890548e6_01a9_4d90_a6b3_781b15f8ec79.slice/crio-c0075f1c0d5d57ebaf3a25cb5c1ca6743869b2765e91138a1e0db94dc2bc76b4 WatchSource:0}: Error finding container c0075f1c0d5d57ebaf3a25cb5c1ca6743869b2765e91138a1e0db94dc2bc76b4: Status 404 returned error can't find the container with id c0075f1c0d5d57ebaf3a25cb5c1ca6743869b2765e91138a1e0db94dc2bc76b4 Jul 22 18:45:02.909070 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:02.908997 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgnhk" event={"ID":"c44ab18a-bae2-4048-a505-bb14abbe90b7","Type":"ContainerStarted","Data":"01f72f5007e12d94e3417c10696cc12947b751c095ec926aa917ef7dd5f94c28"} Jul 22 18:45:02.910044 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:02.910020 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" event={"ID":"890548e6-01a9-4d90-a6b3-781b15f8ec79","Type":"ContainerStarted","Data":"c0075f1c0d5d57ebaf3a25cb5c1ca6743869b2765e91138a1e0db94dc2bc76b4"} Jul 22 18:45:04.334917 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:04.334877 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:04.447446 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:04.447414 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:06.758643 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:06.758572 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:06.925658 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:06.925627 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgnhk" event={"ID":"c44ab18a-bae2-4048-a505-bb14abbe90b7","Type":"ContainerStarted","Data":"5ed467138358806c9e5b30bac99f6b5fc1e8e4ad1a16f9d8cbae4dfaf038b9a3"} Jul 22 18:45:06.927185 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:06.927161 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" event={"ID":"890548e6-01a9-4d90-a6b3-781b15f8ec79","Type":"ContainerStarted","Data":"fa46e6ab4f803697eb20d45c41c43003d4ec43abfe92594ce3f2b0f0b413317e"} Jul 22 18:45:06.927294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:06.927203 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" event={"ID":"890548e6-01a9-4d90-a6b3-781b15f8ec79","Type":"ContainerStarted","Data":"0486d68dc01658a9abd8f093ae14c1dab9ebeca70751601e2104b76a7c755de8"} Jul 22 18:45:06.945352 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:06.945311 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-hxjw6" podStartSLOduration=4.042447843 podStartE2EDuration="7.94529806s" totalImagesPullingTime="3.902850217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:45:06.944350313 +0000 UTC m=+317.391745579" watchObservedRunningTime="2026-07-22 18:45:06.94529806 +0000 UTC m=+317.392693291" Jul 22 18:45:07.985248 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:07.985216 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:08.070195 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:08.070158 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:08.724578 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:08.724532 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:08.824354 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:08.824323 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:08.924957 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:08.924908 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:09.043841 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.043750 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:09.164596 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.164523 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:09.220585 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.220518 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jtlfw"] Jul 22 18:45:09.225842 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.225819 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.228246 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.228220 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4hqlz\"" Jul 22 18:45:09.228459 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.228444 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 22 18:45:09.228619 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.228597 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 22 18:45:09.228702 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.228607 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 22 18:45:09.239446 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.239383 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jtlfw"] Jul 22 18:45:09.244886 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.244806 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q2pth"] Jul 22 18:45:09.249454 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.249432 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.251868 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.251629 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bns8p\"" Jul 22 18:45:09.251868 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.251685 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 18:45:09.252311 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.252140 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 18:45:09.252882 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.252710 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 18:45:09.266574 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.263014 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:09.289084 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.288785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xg8\" (UniqueName: \"kubernetes.io/projected/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-api-access-n5xg8\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.289084 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.288842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.289084 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.288867 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-textfile\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.289084 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.288894 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/070aea45-a91b-4d2b-8883-11c40e0086ce-root\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.289084 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.288935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.289084 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.289001 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmmbr\" (UniqueName: \"kubernetes.io/projected/070aea45-a91b-4d2b-8883-11c40e0086ce-kube-api-access-dmmbr\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.289084 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.289042 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.289084 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.289066 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/070aea45-a91b-4d2b-8883-11c40e0086ce-sys\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.289553 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.289110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-wtmp\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.289553 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.289160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.289553 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.289192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-tls\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.289553 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.289226 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/070aea45-a91b-4d2b-8883-11c40e0086ce-metrics-client-ca\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.289553 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.289254 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.289553 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.289326 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.289553 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.289368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.390534 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390461 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-wtmp\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.390534 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390528 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.390730 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-tls\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.390730 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/070aea45-a91b-4d2b-8883-11c40e0086ce-metrics-client-ca\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.390730 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390616 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.390730 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.390730 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390666 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-wtmp\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.390730 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.391025 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xg8\" (UniqueName: \"kubernetes.io/projected/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-api-access-n5xg8\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.391025 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:45:09.390764 2574 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 22 18:45:09.391025 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390776 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.391025 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-textfile\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.391025 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:45:09.390816 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-state-metrics-tls podName:9a95b2ad-2f10-4864-9402-7ca7c1a930bd nodeName:}" failed. No retries permitted until 2026-07-22 18:45:09.890802527 +0000 UTC m=+320.338197741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-jtlfw" (UID: "9a95b2ad-2f10-4864-9402-7ca7c1a930bd") : secret "kube-state-metrics-tls" not found Jul 22 18:45:09.391025 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/070aea45-a91b-4d2b-8883-11c40e0086ce-root\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.391025 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390879 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.391025 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmmbr\" (UniqueName: \"kubernetes.io/projected/070aea45-a91b-4d2b-8883-11c40e0086ce-kube-api-access-dmmbr\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.391025 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.390954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.391025 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.391011 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/070aea45-a91b-4d2b-8883-11c40e0086ce-sys\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.391499 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.391110 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/070aea45-a91b-4d2b-8883-11c40e0086ce-sys\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.391499 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.391176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.391499 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.391271 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/070aea45-a91b-4d2b-8883-11c40e0086ce-root\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.391643 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.391575 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.392263 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.391811 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.392263 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.391819 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-textfile\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.392263 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.392248 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/070aea45-a91b-4d2b-8883-11c40e0086ce-metrics-client-ca\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.392690 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.392662 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.393663 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.393642 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-tls\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.394722 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.394700 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.394722 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.394717 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/070aea45-a91b-4d2b-8883-11c40e0086ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.403471 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.403427 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmmbr\" (UniqueName: \"kubernetes.io/projected/070aea45-a91b-4d2b-8883-11c40e0086ce-kube-api-access-dmmbr\") pod \"node-exporter-q2pth\" (UID: \"070aea45-a91b-4d2b-8883-11c40e0086ce\") " pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.409866 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.408854 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xg8\" (UniqueName: \"kubernetes.io/projected/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-api-access-n5xg8\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.418667 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.418641 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:09.513239 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.513204 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:09.560249 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.560221 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q2pth" Jul 22 18:45:09.644189 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.644098 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:09.895441 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.895363 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.898069 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.898043 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a95b2ad-2f10-4864-9402-7ca7c1a930bd-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jtlfw\" (UID: \"9a95b2ad-2f10-4864-9402-7ca7c1a930bd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:09.914282 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:09.914256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:10.115401 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:10.115296 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:10.143590 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:10.143352 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" Jul 22 18:45:10.209080 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:10.209034 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:10.280423 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:10.280387 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:10.409140 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:10.409050 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:10.635611 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:45:10.635569 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070aea45_a91b_4d2b_8883_11c40e0086ce.slice/crio-d81c0fecab508f49f2ac0f4d62d4b0922fed7303bb695466e33cdf5d62d030cf WatchSource:0}: Error finding container d81c0fecab508f49f2ac0f4d62d4b0922fed7303bb695466e33cdf5d62d030cf: Status 404 returned error can't find the container with id d81c0fecab508f49f2ac0f4d62d4b0922fed7303bb695466e33cdf5d62d030cf Jul 22 18:45:10.638408 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:10.638277 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-97cd48d98-2wkxk"] Jul 22 18:45:10.745376 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:10.745245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:10.767405 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:10.767384 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jtlfw"] Jul 22 18:45:10.769639 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:45:10.769616 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a95b2ad_2f10_4864_9402_7ca7c1a930bd.slice/crio-25288ea55f2f37d4fdb3a0d7d7982c9a99b0d4b44e39cbe812cdd78fdc7b43de WatchSource:0}: Error finding container 25288ea55f2f37d4fdb3a0d7d7982c9a99b0d4b44e39cbe812cdd78fdc7b43de: Status 404 returned error can't find the container with id 25288ea55f2f37d4fdb3a0d7d7982c9a99b0d4b44e39cbe812cdd78fdc7b43de Jul 22 18:45:10.845018 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:10.844993 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:10.943598 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:10.943524 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" event={"ID":"9a95b2ad-2f10-4864-9402-7ca7c1a930bd","Type":"ContainerStarted","Data":"25288ea55f2f37d4fdb3a0d7d7982c9a99b0d4b44e39cbe812cdd78fdc7b43de"} Jul 22 18:45:10.945493 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:10.945467 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgnhk" event={"ID":"c44ab18a-bae2-4048-a505-bb14abbe90b7","Type":"ContainerStarted","Data":"f617442e9ce52742ca3a13b03699f6c4b2ed172558cb48c17bb7fc5188d6884f"} Jul 22 18:45:10.946552 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:10.946530 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2pth" event={"ID":"070aea45-a91b-4d2b-8883-11c40e0086ce","Type":"ContainerStarted","Data":"d81c0fecab508f49f2ac0f4d62d4b0922fed7303bb695466e33cdf5d62d030cf"} Jul 22 18:45:10.963714 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:10.963672 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hgnhk" podStartSLOduration=32.645321344 podStartE2EDuration="45.963657292s" totalImagesPullingTime="13.318335948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:44:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:45:10.962200964 +0000 UTC m=+321.409596197" watchObservedRunningTime="2026-07-22 18:45:10.963657292 +0000 UTC m=+321.411052523" Jul 22 18:45:11.078210 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:11.078174 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:11.176902 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:11.176874 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:11.951744 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:11.951715 2574 generic.go:356] "Generic (PLEG): container finished" podID="070aea45-a91b-4d2b-8883-11c40e0086ce" containerID="eb773d9033cc6ab8b0caa0d3d7d0eb3c02815330c58f8a47725b8a4df6d18d79" exitCode=0 Jul 22 18:45:11.951878 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:11.951791 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2pth" event={"ID":"070aea45-a91b-4d2b-8883-11c40e0086ce","Type":"ContainerDied","Data":"eb773d9033cc6ab8b0caa0d3d7d0eb3c02815330c58f8a47725b8a4df6d18d79"} Jul 22 18:45:12.081384 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:12.081349 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:12.165115 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:12.165080 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:12.326937 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:12.326908 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:12.487865 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:12.487837 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:12.958318 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:12.958287 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2pth" event={"ID":"070aea45-a91b-4d2b-8883-11c40e0086ce","Type":"ContainerStarted","Data":"86fca888af6a39a124e1afd58f6332d4dd66a560ef0d2c64dd82bf97f6978d1f"} Jul 22 18:45:12.958484 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:12.958334 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2pth" event={"ID":"070aea45-a91b-4d2b-8883-11c40e0086ce","Type":"ContainerStarted","Data":"944a0d6efcc8fc32429d19df0e9174a72b3aa9fb0dd04172c9445fe83fc977ef"} Jul 22 18:45:12.982019 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:12.981949 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q2pth" podStartSLOduration=3.109064672 podStartE2EDuration="3.981935115s" totalImagesPullingTime="808.873832ms" totalInitContainerRuntime="63.996611ms" isStatefulPod=true podCreationTimestamp="2026-07-22 18:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:45:12.980469664 +0000 UTC m=+323.427864896" watchObservedRunningTime="2026-07-22 18:45:12.981935115 +0000 UTC m=+323.429330347" Jul 22 18:45:13.937107 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:13.937055 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:14.009964 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:14.009937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:14.196411 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:14.196329 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:14.968077 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:14.967995 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" event={"ID":"9a95b2ad-2f10-4864-9402-7ca7c1a930bd","Type":"ContainerStarted","Data":"fdbf12e614360a47fc4bb7d4f629ebbf542fe9344942f763253af3efd8ade993"} Jul 22 18:45:14.968077 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:14.968061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" event={"ID":"9a95b2ad-2f10-4864-9402-7ca7c1a930bd","Type":"ContainerStarted","Data":"7545d09aad063f19b0d58bb00ef3774fd8309f51861653aa112b344ff74d2e5f"} Jul 22 18:45:14.968558 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:14.968092 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" event={"ID":"9a95b2ad-2f10-4864-9402-7ca7c1a930bd","Type":"ContainerStarted","Data":"e37d192bc03014d5ad7d55adee10d1d1fa1d71c88eb14a18f22307a74f06ca81"} Jul 22 18:45:14.986083 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:14.986039 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtlfw" podStartSLOduration=2.055785301 podStartE2EDuration="5.986025306s" totalImagesPullingTime="3.930240005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:45:14.984533138 +0000 UTC m=+325.431928372" watchObservedRunningTime="2026-07-22 18:45:14.986025306 +0000 UTC m=+325.433420538" Jul 22 18:45:15.330285 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.330242 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:15.340744 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.340714 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 18:45:15.346097 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.346076 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.348282 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.348254 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 22 18:45:15.348656 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.348482 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6bdfm\"" Jul 22 18:45:15.348775 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.348754 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 22 18:45:15.348832 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.348807 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 22 18:45:15.350428 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.349302 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 18:45:15.350428 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.349616 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 22 18:45:15.350428 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.349632 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 22 18:45:15.350428 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.350159 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cah9upl5nvtio\"" Jul 22 18:45:15.350664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.350435 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 22 18:45:15.350664 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.350598 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 22 18:45:15.350751 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.350679 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 22 18:45:15.351028 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.350879 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 22 18:45:15.351028 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.350892 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 22 18:45:15.351028 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.350927 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 22 18:45:15.351274 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.351121 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 22 18:45:15.353005 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.352958 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 22 18:45:15.355181 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.355163 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 22 18:45:15.363301 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.363221 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 18:45:15.441074 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441035 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441240 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441103 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441240 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441240 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441240 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441184 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441240 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441229 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441485 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441257 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-config-out\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441485 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441278 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441485 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441302 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441485 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441326 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441485 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441352 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441485 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441374 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441485 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441403 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-web-config\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441485 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441485 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441447 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441485 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-config\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441901 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441514 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441901 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441541 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441901 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441573 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.441901 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.441595 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t7dm\" (UniqueName: \"kubernetes.io/projected/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-kube-api-access-4t7dm\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.541965 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.541934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-config\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542155 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.541987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542155 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542155 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542155 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t7dm\" (UniqueName: \"kubernetes.io/projected/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-kube-api-access-4t7dm\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542155 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542155 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542118 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542155 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542504 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542504 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542203 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542504 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542504 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-config-out\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542504 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542504 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542325 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542504 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542504 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542370 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542504 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542390 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542504 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542419 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-web-config\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542504 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542444 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.542504 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.543093 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.543093 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.542936 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.543210 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.543102 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.545955 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.543723 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.545955 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.544378 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.545955 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.545461 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.545955 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.545543 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.546239 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.546084 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-config\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.546239 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.546174 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.547915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.546599 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.547915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.546752 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.547915 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.547156 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.548254 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.548209 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.548704 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.548484 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.548704 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.548671 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-config-out\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.549025 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.549002 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.549776 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.549393 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.549871 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.549743 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-web-config\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.551000 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.550964 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.552840 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.552819 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t7dm\" (UniqueName: \"kubernetes.io/projected/7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5-kube-api-access-4t7dm\") pod \"prometheus-k8s-0\" (UID: \"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.657552 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.657473 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:15.808880 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.808858 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 18:45:15.810667 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:45:15.810629 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dcf2bbc_8811_4e3b_bb53_ad8dd34213c5.slice/crio-98c3e29cf15a8207445934f56d2ba9a2919b04e2015b2f9698677b8e10c5cc61 WatchSource:0}: Error finding container 98c3e29cf15a8207445934f56d2ba9a2919b04e2015b2f9698677b8e10c5cc61: Status 404 returned error can't find the container with id 98c3e29cf15a8207445934f56d2ba9a2919b04e2015b2f9698677b8e10c5cc61 Jul 22 18:45:15.942900 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.942827 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:15.972543 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:15.972513 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5","Type":"ContainerStarted","Data":"98c3e29cf15a8207445934f56d2ba9a2919b04e2015b2f9698677b8e10c5cc61"} Jul 22 18:45:16.003956 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:16.003917 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:16.798757 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:16.798720 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:16.860015 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:16.859990 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:18.737215 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:18.737185 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:19.986830 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:19.986798 2574 generic.go:356] "Generic (PLEG): container finished" podID="7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5" containerID="c4050fa1f3fa45e141825f63d4060a26f1573e8bda91a8dbbf164a1dd651d7f9" exitCode=0 Jul 22 18:45:19.987193 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:19.986880 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5","Type":"ContainerDied","Data":"c4050fa1f3fa45e141825f63d4060a26f1573e8bda91a8dbbf164a1dd651d7f9"} Jul 22 18:45:23.111629 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:23.111580 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:23.170453 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:23.170423 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:24.006062 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:24.006033 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5","Type":"ContainerStarted","Data":"42898809c375fb6d89878659d79221b3348c7cefc125b27fd8c32f769ab17447"} Jul 22 18:45:24.006225 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:24.006093 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5","Type":"ContainerStarted","Data":"fcaa2ea2e277b7d542d4083f8338f3647d4d63f69bfe5ec93d855dc8c5abafa3"} Jul 22 18:45:25.301223 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:25.301178 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:25.401391 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:25.401352 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:25.500421 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:25.500385 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:26.019394 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:26.018045 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5","Type":"ContainerStarted","Data":"8f4910cd1172085edfae1afd038bd36eb96dbc888f272dc30e3367e86535de3f"} Jul 22 18:45:26.019692 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:26.019639 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5","Type":"ContainerStarted","Data":"8844e72085962bf3a6012144d0a93cbb985e1382a1b7f984ba63f96e25cc363a"} Jul 22 18:45:26.019822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:26.019704 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5","Type":"ContainerStarted","Data":"84928fd6e103469cf31ee28713bed7fdc23436abcc066d681db4286671eacd53"} Jul 22 18:45:26.019822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:26.019750 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5","Type":"ContainerStarted","Data":"0c8b6c61af18cec8087aa41f604fcebc633d362a5cf5f57cf3e394a7d98e8456"} Jul 22 18:45:26.048202 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:26.048134 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.928825663 podStartE2EDuration="11.048112265s" totalImagesPullingTime="9.101293107s" totalInitContainerRuntime="17.993495ms" isStatefulPod=false podCreationTimestamp="2026-07-22 18:45:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:45:26.047034158 +0000 UTC m=+336.494429390" watchObservedRunningTime="2026-07-22 18:45:26.048112265 +0000 UTC m=+336.495507509" Jul 22 18:45:30.658296 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:30.658264 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:45:31.807024 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:31.806964 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:31.900110 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:31.900080 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:35.660498 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:35.660440 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" podUID="3f689b8a-ab35-4f36-a20d-047e5614eac1" containerName="registry" containerID="cri-o://8daa6750e537e39d1bcc84537e115a669d3c6120822f3a6b1b0375e8f85710a8" gracePeriod=30 Jul 22 18:45:36.053967 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.053935 2574 generic.go:356] "Generic (PLEG): container finished" podID="3f689b8a-ab35-4f36-a20d-047e5614eac1" containerID="8daa6750e537e39d1bcc84537e115a669d3c6120822f3a6b1b0375e8f85710a8" exitCode=0 Jul 22 18:45:36.054139 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.054007 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" event={"ID":"3f689b8a-ab35-4f36-a20d-047e5614eac1","Type":"ContainerDied","Data":"8daa6750e537e39d1bcc84537e115a669d3c6120822f3a6b1b0375e8f85710a8"} Jul 22 18:45:36.446504 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.446481 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:45:36.515125 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.515100 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f689b8a-ab35-4f36-a20d-047e5614eac1-image-registry-private-configuration\") pod \"3f689b8a-ab35-4f36-a20d-047e5614eac1\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " Jul 22 18:45:36.515288 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.515139 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-bound-sa-token\") pod \"3f689b8a-ab35-4f36-a20d-047e5614eac1\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " Jul 22 18:45:36.515288 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.515171 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f689b8a-ab35-4f36-a20d-047e5614eac1-installation-pull-secrets\") pod \"3f689b8a-ab35-4f36-a20d-047e5614eac1\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " Jul 22 18:45:36.515288 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.515209 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-certificates\") pod \"3f689b8a-ab35-4f36-a20d-047e5614eac1\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " Jul 22 18:45:36.515288 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.515243 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f689b8a-ab35-4f36-a20d-047e5614eac1-ca-trust-extracted\") pod \"3f689b8a-ab35-4f36-a20d-047e5614eac1\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " Jul 22 18:45:36.515288 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.515268 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f689b8a-ab35-4f36-a20d-047e5614eac1-trusted-ca\") pod \"3f689b8a-ab35-4f36-a20d-047e5614eac1\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " Jul 22 18:45:36.515532 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.515356 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt85f\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-kube-api-access-mt85f\") pod \"3f689b8a-ab35-4f36-a20d-047e5614eac1\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " Jul 22 18:45:36.515532 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.515385 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") pod \"3f689b8a-ab35-4f36-a20d-047e5614eac1\" (UID: \"3f689b8a-ab35-4f36-a20d-047e5614eac1\") " Jul 22 18:45:36.515679 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.515648 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3f689b8a-ab35-4f36-a20d-047e5614eac1" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 18:45:36.515743 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.515701 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f689b8a-ab35-4f36-a20d-047e5614eac1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3f689b8a-ab35-4f36-a20d-047e5614eac1" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 18:45:36.517832 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.517733 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3f689b8a-ab35-4f36-a20d-047e5614eac1" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 18:45:36.517953 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.517931 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f689b8a-ab35-4f36-a20d-047e5614eac1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3f689b8a-ab35-4f36-a20d-047e5614eac1" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 18:45:36.518086 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.518062 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f689b8a-ab35-4f36-a20d-047e5614eac1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3f689b8a-ab35-4f36-a20d-047e5614eac1" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 18:45:36.518156 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.518134 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-kube-api-access-mt85f" (OuterVolumeSpecName: "kube-api-access-mt85f") pod "3f689b8a-ab35-4f36-a20d-047e5614eac1" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1"). InnerVolumeSpecName "kube-api-access-mt85f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 18:45:36.518194 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.518176 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3f689b8a-ab35-4f36-a20d-047e5614eac1" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 18:45:36.524642 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.524616 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f689b8a-ab35-4f36-a20d-047e5614eac1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3f689b8a-ab35-4f36-a20d-047e5614eac1" (UID: "3f689b8a-ab35-4f36-a20d-047e5614eac1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 18:45:36.616675 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.616590 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt85f\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-kube-api-access-mt85f\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 18:45:36.616675 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.616622 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 18:45:36.616675 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.616636 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f689b8a-ab35-4f36-a20d-047e5614eac1-image-registry-private-configuration\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 18:45:36.616675 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.616649 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f689b8a-ab35-4f36-a20d-047e5614eac1-bound-sa-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 18:45:36.616675 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.616662 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f689b8a-ab35-4f36-a20d-047e5614eac1-installation-pull-secrets\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 18:45:36.616675 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.616674 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f689b8a-ab35-4f36-a20d-047e5614eac1-registry-certificates\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 18:45:36.616962 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.616687 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f689b8a-ab35-4f36-a20d-047e5614eac1-ca-trust-extracted\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 18:45:36.616962 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:36.616699 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f689b8a-ab35-4f36-a20d-047e5614eac1-trusted-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 18:45:37.059108 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:37.059082 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" Jul 22 18:45:37.059108 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:37.059100 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97cd48d98-2wkxk" event={"ID":"3f689b8a-ab35-4f36-a20d-047e5614eac1","Type":"ContainerDied","Data":"6029aefaaa59dbee6ba89d586267395107d479bcef254666741648bbc315ffcf"} Jul 22 18:45:37.059561 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:37.059146 2574 scope.go:117] "RemoveContainer" containerID="8daa6750e537e39d1bcc84537e115a669d3c6120822f3a6b1b0375e8f85710a8" Jul 22 18:45:37.079031 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:37.079008 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-97cd48d98-2wkxk"] Jul 22 18:45:37.082654 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:37.082633 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-97cd48d98-2wkxk"] Jul 22 18:45:38.004215 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:38.004180 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f689b8a-ab35-4f36-a20d-047e5614eac1" path="/var/lib/kubelet/pods/3f689b8a-ab35-4f36-a20d-047e5614eac1/volumes" Jul 22 18:45:45.973111 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:45.973025 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8ph6g_7fe1adde-5816-463f-aa6a-7769d7569b7f/serve-healthcheck-canary/0.log" Jul 22 18:45:46.810179 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:46.810138 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:46.881114 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:46.881082 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:54.163288 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:54.163253 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:54.300488 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:54.300451 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:45:54.391913 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:45:54.391881 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:03.835755 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:03.835709 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:46:03.837935 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:03.837913 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27e81e3-a9d6-4c4b-b138-3a4394204eb3-metrics-certs\") pod \"network-metrics-daemon-lc7qt\" (UID: \"d27e81e3-a9d6-4c4b-b138-3a4394204eb3\") " pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:46:04.103019 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:04.102909 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g2vpr\"" Jul 22 18:46:04.110686 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:04.110663 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7qt" Jul 22 18:46:04.243561 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:04.243533 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lc7qt"] Jul 22 18:46:04.246038 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:46:04.246013 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27e81e3_a9d6_4c4b_b138_3a4394204eb3.slice/crio-fddccdb162a7b460f6407eb4e0b1ed285555b7d8a837e9ef978eb77e64613c23 WatchSource:0}: Error finding container fddccdb162a7b460f6407eb4e0b1ed285555b7d8a837e9ef978eb77e64613c23: Status 404 returned error can't find the container with id fddccdb162a7b460f6407eb4e0b1ed285555b7d8a837e9ef978eb77e64613c23 Jul 22 18:46:05.164581 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:05.164547 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lc7qt" event={"ID":"d27e81e3-a9d6-4c4b-b138-3a4394204eb3","Type":"ContainerStarted","Data":"fddccdb162a7b460f6407eb4e0b1ed285555b7d8a837e9ef978eb77e64613c23"} Jul 22 18:46:06.169663 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:06.169623 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lc7qt" event={"ID":"d27e81e3-a9d6-4c4b-b138-3a4394204eb3","Type":"ContainerStarted","Data":"302a33079ee9ad75e35cb3cac240cd892f7c3d90f5cae27e5b9a15a2b2d314f4"} Jul 22 18:46:06.170073 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:06.169672 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lc7qt" event={"ID":"d27e81e3-a9d6-4c4b-b138-3a4394204eb3","Type":"ContainerStarted","Data":"cd12dfed1dc02f59c317bb1e1675541bb390774efedc5f08992f08eb24003501"} Jul 22 18:46:06.190252 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:06.190195 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lc7qt" podStartSLOduration=375.125605361 podStartE2EDuration="6m16.190178034s" totalImagesPullingTime="1.064572673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:46:06.189004737 +0000 UTC m=+376.636399972" watchObservedRunningTime="2026-07-22 18:46:06.190178034 +0000 UTC m=+376.637573270" Jul 22 18:46:06.310147 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:06.310117 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:06.372778 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:06.372751 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:08.647131 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:08.647102 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:08.748158 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:08.748122 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:15.657824 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:15.657791 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:46:15.676553 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:15.676527 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:46:16.220163 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:16.220124 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 18:46:17.043035 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:17.042991 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:17.128204 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:17.128176 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:28.610448 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:28.610404 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:28.974182 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:28.974150 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:35.794800 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:46:35.794764 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" podUID="b117ac7f-9bc8-46e2-a049-fdc145a52523" Jul 22 18:46:36.281760 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:36.281735 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:46:37.873429 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:37.873394 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:46:37.875719 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:37.875696 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b117ac7f-9bc8-46e2-a049-fdc145a52523-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-78xp8\" (UID: \"b117ac7f-9bc8-46e2-a049-fdc145a52523\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:46:38.086031 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:38.085993 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7492w\"" Jul 22 18:46:38.093963 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:38.093936 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" Jul 22 18:46:38.294385 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:38.294354 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8"] Jul 22 18:46:38.297528 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:46:38.297505 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb117ac7f_9bc8_46e2_a049_fdc145a52523.slice/crio-adbe10e27969f3d5501fc819d438414378f41955572afd7ab7fb12ffca30cef6 WatchSource:0}: Error finding container adbe10e27969f3d5501fc819d438414378f41955572afd7ab7fb12ffca30cef6: Status 404 returned error can't find the container with id adbe10e27969f3d5501fc819d438414378f41955572afd7ab7fb12ffca30cef6 Jul 22 18:46:39.293817 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:39.293785 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" event={"ID":"b117ac7f-9bc8-46e2-a049-fdc145a52523","Type":"ContainerStarted","Data":"adbe10e27969f3d5501fc819d438414378f41955572afd7ab7fb12ffca30cef6"} Jul 22 18:46:39.360156 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:39.360122 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:39.492675 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:39.492651 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:40.299530 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:40.299495 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" event={"ID":"b117ac7f-9bc8-46e2-a049-fdc145a52523","Type":"ContainerStarted","Data":"c79a64653a1524aa482218fd994227f77a9ff5083afd9ece75fc6898a4787cd6"} Jul 22 18:46:40.319429 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:40.319367 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-78xp8" podStartSLOduration=392.150263421 podStartE2EDuration="6m33.319352228s" totalImagesPullingTime="1.169088807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:46:40.318582335 +0000 UTC m=+410.765977567" watchObservedRunningTime="2026-07-22 18:46:40.319352228 +0000 UTC m=+410.766747460" Jul 22 18:46:40.466966 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:40.466933 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:40.573339 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:40.573265 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:43.950199 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:43.950151 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:44.014648 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:44.014621 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:47.350469 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:47.350438 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:47.450095 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:47.450065 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:57.220496 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:57.220455 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:46:57.295603 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:46:57.295570 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:47:01.866064 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:47:01.866031 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:47:01.966281 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:47:01.966245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:47:10.498822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:47:10.498780 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:47:16.186287 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:47:16.186210 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:47:16.287876 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:47:16.287835 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:47:36.570030 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:47:36.569991 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:47:55.287076 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:47:55.287041 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:06.795791 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:06.795748 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:06.892420 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:06.892394 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:35.092827 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:35.092792 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:35.193205 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:35.193160 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:37.105526 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:37.105496 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:37.217931 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:37.217899 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:38.021043 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:38.021011 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:38.116490 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:38.116460 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:38.408833 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:38.408764 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:38.535390 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:38.535358 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:40.213749 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:40.213714 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:40.820923 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:40.820890 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:41.054495 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.054466 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ms972"] Jul 22 18:48:41.054773 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.054760 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f689b8a-ab35-4f36-a20d-047e5614eac1" containerName="registry" Jul 22 18:48:41.054827 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.054775 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f689b8a-ab35-4f36-a20d-047e5614eac1" containerName="registry" Jul 22 18:48:41.054827 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.054825 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f689b8a-ab35-4f36-a20d-047e5614eac1" containerName="registry" Jul 22 18:48:41.057567 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.057550 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ms972" Jul 22 18:48:41.060010 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.059982 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 18:48:41.060010 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.059964 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 18:48:41.060187 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.060068 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nhmh5\"" Jul 22 18:48:41.065090 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.065066 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ms972"] Jul 22 18:48:41.172902 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.172844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25349a74-425e-4b25-b988-b25adfbbfab2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ms972\" (UID: \"25349a74-425e-4b25-b988-b25adfbbfab2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ms972" Jul 22 18:48:41.172902 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.172875 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m4ms\" (UniqueName: \"kubernetes.io/projected/25349a74-425e-4b25-b988-b25adfbbfab2-kube-api-access-8m4ms\") pod \"cert-manager-cainjector-559cdd8c87-ms972\" (UID: \"25349a74-425e-4b25-b988-b25adfbbfab2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ms972" Jul 22 18:48:41.273753 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.273714 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25349a74-425e-4b25-b988-b25adfbbfab2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ms972\" (UID: \"25349a74-425e-4b25-b988-b25adfbbfab2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ms972" Jul 22 18:48:41.273753 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.273760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m4ms\" (UniqueName: \"kubernetes.io/projected/25349a74-425e-4b25-b988-b25adfbbfab2-kube-api-access-8m4ms\") pod \"cert-manager-cainjector-559cdd8c87-ms972\" (UID: \"25349a74-425e-4b25-b988-b25adfbbfab2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ms972" Jul 22 18:48:41.281803 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.281778 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25349a74-425e-4b25-b988-b25adfbbfab2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ms972\" (UID: \"25349a74-425e-4b25-b988-b25adfbbfab2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ms972" Jul 22 18:48:41.281938 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.281921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m4ms\" (UniqueName: \"kubernetes.io/projected/25349a74-425e-4b25-b988-b25adfbbfab2-kube-api-access-8m4ms\") pod \"cert-manager-cainjector-559cdd8c87-ms972\" (UID: \"25349a74-425e-4b25-b988-b25adfbbfab2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ms972" Jul 22 18:48:41.377223 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.377191 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ms972" Jul 22 18:48:41.504643 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.504618 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ms972"] Jul 22 18:48:41.507196 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:48:41.507167 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25349a74_425e_4b25_b988_b25adfbbfab2.slice/crio-c81a599f0f03ac627df26081135ca334bb08ebba66e444bdc1857b3e6bc8e214 WatchSource:0}: Error finding container c81a599f0f03ac627df26081135ca334bb08ebba66e444bdc1857b3e6bc8e214: Status 404 returned error can't find the container with id c81a599f0f03ac627df26081135ca334bb08ebba66e444bdc1857b3e6bc8e214 Jul 22 18:48:41.749775 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:41.749736 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ms972" event={"ID":"25349a74-425e-4b25-b988-b25adfbbfab2","Type":"ContainerStarted","Data":"c81a599f0f03ac627df26081135ca334bb08ebba66e444bdc1857b3e6bc8e214"} Jul 22 18:48:43.492689 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:43.492597 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:43.593070 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:43.593016 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:44.764377 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:44.764291 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ms972" event={"ID":"25349a74-425e-4b25-b988-b25adfbbfab2","Type":"ContainerStarted","Data":"ea821f3444f6bc30bf2dd4a69e4534b1684d6293c4ce2180b55819386d4a3270"} Jul 22 18:48:44.782725 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:44.782668 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ms972" podStartSLOduration=0.809865711 podStartE2EDuration="3.782654441s" totalImagesPullingTime="2.97278873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:48:44.781807289 +0000 UTC m=+535.229202522" watchObservedRunningTime="2026-07-22 18:48:44.782654441 +0000 UTC m=+535.230049673" Jul 22 18:48:48.618815 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:48.618781 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:50.572876 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:50.572839 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:52.829489 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:52.829455 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:59.187861 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:59.187825 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:48:59.286872 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:48:59.286839 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:00.930412 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:00.930371 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:11.307390 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:11.307357 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:11.369206 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:11.369179 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:16.928623 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:16.928592 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:17.022031 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.022001 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:17.135363 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.135332 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:17.222404 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.222382 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw"] Jul 22 18:49:17.225588 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.225570 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.229149 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.229127 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 18:49:17.229286 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.229225 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 18:49:17.229286 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.229256 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 18:49:17.229286 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.229267 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 18:49:17.229506 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.229268 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 18:49:17.229747 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.229734 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fx7kj\"" Jul 22 18:49:17.232636 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.232617 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c576f0eb-eb3e-439f-9146-87fd146e8911-metrics-cert\") pod \"lws-controller-manager-5b444cc6d8-rqclw\" (UID: \"c576f0eb-eb3e-439f-9146-87fd146e8911\") " pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.232727 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.232643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c576f0eb-eb3e-439f-9146-87fd146e8911-manager-config\") pod \"lws-controller-manager-5b444cc6d8-rqclw\" (UID: \"c576f0eb-eb3e-439f-9146-87fd146e8911\") " pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.232727 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.232661 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9st9\" (UniqueName: \"kubernetes.io/projected/c576f0eb-eb3e-439f-9146-87fd146e8911-kube-api-access-k9st9\") pod \"lws-controller-manager-5b444cc6d8-rqclw\" (UID: \"c576f0eb-eb3e-439f-9146-87fd146e8911\") " pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.232845 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.232745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c576f0eb-eb3e-439f-9146-87fd146e8911-cert\") pod \"lws-controller-manager-5b444cc6d8-rqclw\" (UID: \"c576f0eb-eb3e-439f-9146-87fd146e8911\") " pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.238371 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.238352 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw"] Jul 22 18:49:17.251984 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.251934 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:17.333386 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.333351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c576f0eb-eb3e-439f-9146-87fd146e8911-metrics-cert\") pod \"lws-controller-manager-5b444cc6d8-rqclw\" (UID: \"c576f0eb-eb3e-439f-9146-87fd146e8911\") " pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.333386 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.333387 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c576f0eb-eb3e-439f-9146-87fd146e8911-manager-config\") pod \"lws-controller-manager-5b444cc6d8-rqclw\" (UID: \"c576f0eb-eb3e-439f-9146-87fd146e8911\") " pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.333634 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.333409 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9st9\" (UniqueName: \"kubernetes.io/projected/c576f0eb-eb3e-439f-9146-87fd146e8911-kube-api-access-k9st9\") pod \"lws-controller-manager-5b444cc6d8-rqclw\" (UID: \"c576f0eb-eb3e-439f-9146-87fd146e8911\") " pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.333634 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.333441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c576f0eb-eb3e-439f-9146-87fd146e8911-cert\") pod \"lws-controller-manager-5b444cc6d8-rqclw\" (UID: \"c576f0eb-eb3e-439f-9146-87fd146e8911\") " pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.334043 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.334011 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c576f0eb-eb3e-439f-9146-87fd146e8911-manager-config\") pod \"lws-controller-manager-5b444cc6d8-rqclw\" (UID: \"c576f0eb-eb3e-439f-9146-87fd146e8911\") " pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.335879 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.335857 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c576f0eb-eb3e-439f-9146-87fd146e8911-metrics-cert\") pod \"lws-controller-manager-5b444cc6d8-rqclw\" (UID: \"c576f0eb-eb3e-439f-9146-87fd146e8911\") " pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.336046 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.335899 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c576f0eb-eb3e-439f-9146-87fd146e8911-cert\") pod \"lws-controller-manager-5b444cc6d8-rqclw\" (UID: \"c576f0eb-eb3e-439f-9146-87fd146e8911\") " pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.341356 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.341334 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9st9\" (UniqueName: \"kubernetes.io/projected/c576f0eb-eb3e-439f-9146-87fd146e8911-kube-api-access-k9st9\") pod \"lws-controller-manager-5b444cc6d8-rqclw\" (UID: \"c576f0eb-eb3e-439f-9146-87fd146e8911\") " pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.534750 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.534662 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:17.633533 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.633477 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:17.664091 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.663939 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw"] Jul 22 18:49:17.666699 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:49:17.666670 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc576f0eb_eb3e_439f_9146_87fd146e8911.slice/crio-1cc3c10e59a2af73e6894095f2e108bc101f8a5191a6c82c6327de9b5c6518e3 WatchSource:0}: Error finding container 1cc3c10e59a2af73e6894095f2e108bc101f8a5191a6c82c6327de9b5c6518e3: Status 404 returned error can't find the container with id 1cc3c10e59a2af73e6894095f2e108bc101f8a5191a6c82c6327de9b5c6518e3 Jul 22 18:49:17.859806 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.859692 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:17.891196 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.891158 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" event={"ID":"c576f0eb-eb3e-439f-9146-87fd146e8911","Type":"ContainerStarted","Data":"1cc3c10e59a2af73e6894095f2e108bc101f8a5191a6c82c6327de9b5c6518e3"} Jul 22 18:49:17.954317 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.954286 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:17.998642 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:17.998608 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc"] Jul 22 18:49:18.003565 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.003541 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.022247 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.022227 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 18:49:18.022374 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.022248 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 18:49:18.023564 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.023547 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bkd9t\"" Jul 22 18:49:18.023643 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.023567 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 18:49:18.023643 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.023612 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 18:49:18.039843 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.039811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a02f7981-bbd1-41d3-bbcd-584bab80e60b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.039982 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.039866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9fv\" (UniqueName: \"kubernetes.io/projected/a02f7981-bbd1-41d3-bbcd-584bab80e60b-kube-api-access-nw9fv\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.039982 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.039912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a02f7981-bbd1-41d3-bbcd-584bab80e60b-manifests\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.040095 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.039984 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a02f7981-bbd1-41d3-bbcd-584bab80e60b-tmp\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.040095 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.040039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a02f7981-bbd1-41d3-bbcd-584bab80e60b-webhook-cert\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.041795 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.041775 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc"] Jul 22 18:49:18.140846 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.140767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a02f7981-bbd1-41d3-bbcd-584bab80e60b-tmp\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.140846 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.140817 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a02f7981-bbd1-41d3-bbcd-584bab80e60b-webhook-cert\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.140846 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.140845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a02f7981-bbd1-41d3-bbcd-584bab80e60b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.141073 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.140868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9fv\" (UniqueName: \"kubernetes.io/projected/a02f7981-bbd1-41d3-bbcd-584bab80e60b-kube-api-access-nw9fv\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.141073 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.140892 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a02f7981-bbd1-41d3-bbcd-584bab80e60b-manifests\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.141305 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.141286 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a02f7981-bbd1-41d3-bbcd-584bab80e60b-manifests\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.143175 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.143154 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a02f7981-bbd1-41d3-bbcd-584bab80e60b-tmp\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.143310 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.143293 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a02f7981-bbd1-41d3-bbcd-584bab80e60b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.143389 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.143373 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a02f7981-bbd1-41d3-bbcd-584bab80e60b-webhook-cert\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.151876 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.151859 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9fv\" (UniqueName: \"kubernetes.io/projected/a02f7981-bbd1-41d3-bbcd-584bab80e60b-kube-api-access-nw9fv\") pod \"opendatahub-operator-controller-manager-f9bb58876-lrjlc\" (UID: \"a02f7981-bbd1-41d3-bbcd-584bab80e60b\") " pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.216080 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.216055 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:18.314709 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.314685 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:18.430500 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.430089 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:18.460924 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.460794 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc"] Jul 22 18:49:18.464044 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:49:18.464019 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02f7981_bbd1_41d3_bbcd_584bab80e60b.slice/crio-df7f6a41847f001371f5c69ed63c4e5ca7e0a7adbffa8d5faab6982bee100e6c WatchSource:0}: Error finding container df7f6a41847f001371f5c69ed63c4e5ca7e0a7adbffa8d5faab6982bee100e6c: Status 404 returned error can't find the container with id df7f6a41847f001371f5c69ed63c4e5ca7e0a7adbffa8d5faab6982bee100e6c Jul 22 18:49:18.640787 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.640741 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:18.764129 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.764095 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:18.898162 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:18.898122 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" event={"ID":"a02f7981-bbd1-41d3-bbcd-584bab80e60b","Type":"ContainerStarted","Data":"df7f6a41847f001371f5c69ed63c4e5ca7e0a7adbffa8d5faab6982bee100e6c"} Jul 22 18:49:22.918436 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:22.918397 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" event={"ID":"c576f0eb-eb3e-439f-9146-87fd146e8911","Type":"ContainerStarted","Data":"ecd584aa0325a19ee6e52bccc05b77050797bb1192715d2d00e42fc962d573b9"} Jul 22 18:49:22.918854 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:22.918466 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:22.920011 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:22.919966 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" event={"ID":"a02f7981-bbd1-41d3-bbcd-584bab80e60b","Type":"ContainerStarted","Data":"8f2a12c685642bea4c0520d9f784737c05798531285c3fd8aa72e225eeec3ed6"} Jul 22 18:49:22.934056 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:22.933991 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" podStartSLOduration=1.678118479 podStartE2EDuration="5.933955818s" totalImagesPullingTime="4.255837339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:49:22.933676977 +0000 UTC m=+573.381072230" watchObservedRunningTime="2026-07-22 18:49:22.933955818 +0000 UTC m=+573.381351051" Jul 22 18:49:23.925884 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:23.925849 2574 generic.go:356] "Generic (PLEG): container finished" podID="a02f7981-bbd1-41d3-bbcd-584bab80e60b" containerID="8f2a12c685642bea4c0520d9f784737c05798531285c3fd8aa72e225eeec3ed6" exitCode=0 Jul 22 18:49:23.926289 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:23.925912 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" event={"ID":"a02f7981-bbd1-41d3-bbcd-584bab80e60b","Type":"ContainerDied","Data":"8f2a12c685642bea4c0520d9f784737c05798531285c3fd8aa72e225eeec3ed6"} Jul 22 18:49:24.932063 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:24.932024 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" event={"ID":"a02f7981-bbd1-41d3-bbcd-584bab80e60b","Type":"ContainerStarted","Data":"d7d926087b4adf33502d22207e583ec0338446853f5ecbea14d8e21ba416fca6"} Jul 22 18:49:24.932589 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:24.932159 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:24.953543 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:24.953495 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" podStartSLOduration=3.071647967 podStartE2EDuration="7.953476878s" totalImagesPullingTime="3.872795446s" totalInitContainerRuntime="1.009033465s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:49:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:49:24.950726895 +0000 UTC m=+575.398122128" watchObservedRunningTime="2026-07-22 18:49:24.953476878 +0000 UTC m=+575.400872111" Jul 22 18:49:33.928643 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:33.928605 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b444cc6d8-rqclw" Jul 22 18:49:34.875783 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:34.875745 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:35.938866 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:35.938835 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f9bb58876-lrjlc" Jul 22 18:49:37.571112 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:37.571078 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:37.695029 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:37.694996 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:43.236411 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.236380 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:43.312120 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.312092 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh"] Jul 22 18:49:43.329809 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.329772 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh"] Jul 22 18:49:43.329950 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.329884 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" Jul 22 18:49:43.332050 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.332007 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 18:49:43.332050 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.332031 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 18:49:43.332340 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.332322 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 18:49:43.332420 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.332326 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 18:49:43.332498 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.332326 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mvb5j\"" Jul 22 18:49:43.440528 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.440504 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e4bf375-301f-4d5d-b0e8-154256b89782-tmp\") pod \"kube-auth-proxy-779ddf9494-gj7dh\" (UID: \"2e4bf375-301f-4d5d-b0e8-154256b89782\") " pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" Jul 22 18:49:43.440671 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.440559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4bf375-301f-4d5d-b0e8-154256b89782-tls-certs\") pod \"kube-auth-proxy-779ddf9494-gj7dh\" (UID: \"2e4bf375-301f-4d5d-b0e8-154256b89782\") " pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" Jul 22 18:49:43.440671 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.440587 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv2h9\" (UniqueName: \"kubernetes.io/projected/2e4bf375-301f-4d5d-b0e8-154256b89782-kube-api-access-qv2h9\") pod \"kube-auth-proxy-779ddf9494-gj7dh\" (UID: \"2e4bf375-301f-4d5d-b0e8-154256b89782\") " pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" Jul 22 18:49:43.533104 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.533043 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:43.541228 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.541204 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e4bf375-301f-4d5d-b0e8-154256b89782-tmp\") pod \"kube-auth-proxy-779ddf9494-gj7dh\" (UID: \"2e4bf375-301f-4d5d-b0e8-154256b89782\") " pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" Jul 22 18:49:43.541346 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.541256 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4bf375-301f-4d5d-b0e8-154256b89782-tls-certs\") pod \"kube-auth-proxy-779ddf9494-gj7dh\" (UID: \"2e4bf375-301f-4d5d-b0e8-154256b89782\") " pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" Jul 22 18:49:43.541346 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.541283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv2h9\" (UniqueName: \"kubernetes.io/projected/2e4bf375-301f-4d5d-b0e8-154256b89782-kube-api-access-qv2h9\") pod \"kube-auth-proxy-779ddf9494-gj7dh\" (UID: \"2e4bf375-301f-4d5d-b0e8-154256b89782\") " pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" Jul 22 18:49:43.541459 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:49:43.541433 2574 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 22 18:49:43.541510 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:49:43.541502 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4bf375-301f-4d5d-b0e8-154256b89782-tls-certs podName:2e4bf375-301f-4d5d-b0e8-154256b89782 nodeName:}" failed. No retries permitted until 2026-07-22 18:49:44.041482896 +0000 UTC m=+594.488878112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2e4bf375-301f-4d5d-b0e8-154256b89782-tls-certs") pod "kube-auth-proxy-779ddf9494-gj7dh" (UID: "2e4bf375-301f-4d5d-b0e8-154256b89782") : secret "kube-auth-proxy-tls" not found Jul 22 18:49:43.543584 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.543564 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e4bf375-301f-4d5d-b0e8-154256b89782-tmp\") pod \"kube-auth-proxy-779ddf9494-gj7dh\" (UID: \"2e4bf375-301f-4d5d-b0e8-154256b89782\") " pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" Jul 22 18:49:43.550013 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:43.549966 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv2h9\" (UniqueName: \"kubernetes.io/projected/2e4bf375-301f-4d5d-b0e8-154256b89782-kube-api-access-qv2h9\") pod \"kube-auth-proxy-779ddf9494-gj7dh\" (UID: \"2e4bf375-301f-4d5d-b0e8-154256b89782\") " pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" Jul 22 18:49:44.046414 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:44.046383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4bf375-301f-4d5d-b0e8-154256b89782-tls-certs\") pod \"kube-auth-proxy-779ddf9494-gj7dh\" (UID: \"2e4bf375-301f-4d5d-b0e8-154256b89782\") " pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" Jul 22 18:49:44.048796 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:44.048765 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4bf375-301f-4d5d-b0e8-154256b89782-tls-certs\") pod \"kube-auth-proxy-779ddf9494-gj7dh\" (UID: \"2e4bf375-301f-4d5d-b0e8-154256b89782\") " pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" Jul 22 18:49:44.239413 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:44.239381 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" Jul 22 18:49:44.384276 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:44.384216 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh"] Jul 22 18:49:44.388231 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:49:44.388205 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4bf375_301f_4d5d_b0e8_154256b89782.slice/crio-1f14fa62bdb85f198bd2f2be434d0e61c473a6287379e327e52193a4518ec593 WatchSource:0}: Error finding container 1f14fa62bdb85f198bd2f2be434d0e61c473a6287379e327e52193a4518ec593: Status 404 returned error can't find the container with id 1f14fa62bdb85f198bd2f2be434d0e61c473a6287379e327e52193a4518ec593 Jul 22 18:49:45.015280 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:45.015230 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" event={"ID":"2e4bf375-301f-4d5d-b0e8-154256b89782","Type":"ContainerStarted","Data":"1f14fa62bdb85f198bd2f2be434d0e61c473a6287379e327e52193a4518ec593"} Jul 22 18:49:45.235874 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:45.235839 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:45.293047 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:45.292962 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:45.556308 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:45.556214 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:48.029544 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:48.029503 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" event={"ID":"2e4bf375-301f-4d5d-b0e8-154256b89782","Type":"ContainerStarted","Data":"dc459b99bd6df95a8418f9258ca2876ac4486e89236c8700478cab08ccf31360"} Jul 22 18:49:48.047713 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:48.047668 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-779ddf9494-gj7dh" podStartSLOduration=2.052513246 podStartE2EDuration="5.047653991s" totalImagesPullingTime="2.995140745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:49:48.046075624 +0000 UTC m=+598.493470855" watchObservedRunningTime="2026-07-22 18:49:48.047653991 +0000 UTC m=+598.495049225" Jul 22 18:49:50.196676 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:50.196637 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:50.297069 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:50.297031 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:50.839553 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:50.839513 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:50.919163 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:50.919125 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:51.111636 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:51.111554 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:51.243141 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:51.243111 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:51.480154 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:51.480120 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:53.319470 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:53.319433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:53.481456 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:53.481392 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:54.842250 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:54.842130 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:54.972618 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:54.972579 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:54.976671 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:54.976647 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-64rcl"] Jul 22 18:49:54.979838 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:54.979818 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-64rcl" Jul 22 18:49:54.984414 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:54.984396 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 22 18:49:54.984706 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:54.984685 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9ljpj\"" Jul 22 18:49:55.017559 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:55.017537 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-64rcl"] Jul 22 18:49:55.129794 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:55.129687 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:55.151300 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:55.151269 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c37545e7-7734-43c0-bfd4-714ed76c147f-cert\") pod \"kserve-controller-manager-d6f4b65b8-64rcl\" (UID: \"c37545e7-7734-43c0-bfd4-714ed76c147f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-64rcl" Jul 22 18:49:55.151458 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:55.151331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv977\" (UniqueName: \"kubernetes.io/projected/c37545e7-7734-43c0-bfd4-714ed76c147f-kube-api-access-rv977\") pod \"kserve-controller-manager-d6f4b65b8-64rcl\" (UID: \"c37545e7-7734-43c0-bfd4-714ed76c147f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-64rcl" Jul 22 18:49:55.218635 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:55.218600 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:55.252857 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:55.252824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c37545e7-7734-43c0-bfd4-714ed76c147f-cert\") pod \"kserve-controller-manager-d6f4b65b8-64rcl\" (UID: \"c37545e7-7734-43c0-bfd4-714ed76c147f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-64rcl" Jul 22 18:49:55.253203 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:55.253179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv977\" (UniqueName: \"kubernetes.io/projected/c37545e7-7734-43c0-bfd4-714ed76c147f-kube-api-access-rv977\") pod \"kserve-controller-manager-d6f4b65b8-64rcl\" (UID: \"c37545e7-7734-43c0-bfd4-714ed76c147f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-64rcl" Jul 22 18:49:55.256435 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:55.256411 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c37545e7-7734-43c0-bfd4-714ed76c147f-cert\") pod \"kserve-controller-manager-d6f4b65b8-64rcl\" (UID: \"c37545e7-7734-43c0-bfd4-714ed76c147f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-64rcl" Jul 22 18:49:55.265234 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:55.265210 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv977\" (UniqueName: \"kubernetes.io/projected/c37545e7-7734-43c0-bfd4-714ed76c147f-kube-api-access-rv977\") pod \"kserve-controller-manager-d6f4b65b8-64rcl\" (UID: \"c37545e7-7734-43c0-bfd4-714ed76c147f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-64rcl" Jul 22 18:49:55.289826 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:55.289800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-64rcl" Jul 22 18:49:55.317171 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:55.317138 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:55.416762 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:55.416736 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-64rcl"] Jul 22 18:49:55.419293 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:49:55.419263 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37545e7_7734_43c0_bfd4_714ed76c147f.slice/crio-74a31bca0f60fccbc87a1595e7ebc85c48483ebc97d21e6c67e5381bca74fc7a WatchSource:0}: Error finding container 74a31bca0f60fccbc87a1595e7ebc85c48483ebc97d21e6c67e5381bca74fc7a: Status 404 returned error can't find the container with id 74a31bca0f60fccbc87a1595e7ebc85c48483ebc97d21e6c67e5381bca74fc7a Jul 22 18:49:56.065395 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:56.065363 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-64rcl" event={"ID":"c37545e7-7734-43c0-bfd4-714ed76c147f","Type":"ContainerStarted","Data":"74a31bca0f60fccbc87a1595e7ebc85c48483ebc97d21e6c67e5381bca74fc7a"} Jul 22 18:49:56.241443 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:56.241407 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:49:56.345092 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:49:56.345023 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:50:00.085085 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:50:00.085040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-64rcl" event={"ID":"c37545e7-7734-43c0-bfd4-714ed76c147f","Type":"ContainerStarted","Data":"333d6d5940c59fe3921caa0501b2a13a989e573938b18b27022e2f50095fceaf"} Jul 22 18:50:00.085468 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:50:00.085120 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-64rcl" Jul 22 18:50:00.103690 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:50:00.103651 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-64rcl" podStartSLOduration=1.937060161 podStartE2EDuration="6.103641198s" totalImagesPullingTime="4.166581037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:50:00.102464395 +0000 UTC m=+610.549859628" watchObservedRunningTime="2026-07-22 18:50:00.103641198 +0000 UTC m=+610.551036430" Jul 22 18:50:31.094146 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:50:31.094110 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-64rcl" Jul 22 18:50:39.176864 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:50:39.176823 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:50:41.693229 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:50:41.693198 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:50:47.383735 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:50:47.383698 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:50:47.484342 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:50:47.484304 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:07.431012 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:07.430959 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:08.092272 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:08.092239 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:09.065949 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:09.065914 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:12.452713 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:12.452675 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:15.251518 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:15.251450 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:15.871840 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:15.871806 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:17.630210 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:17.630172 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:18.107215 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:18.107175 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:21.097541 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:21.097482 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:22.845818 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:22.845779 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:22.944377 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:22.944350 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:23.242337 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:23.242306 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:23.343370 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:23.343337 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:26.697461 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:26.697425 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:26.830995 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:26.830943 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:30.049869 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:30.049792 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:30.147277 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:30.147243 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:31.181164 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:31.181132 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:31.513877 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:31.513841 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:33.976171 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:33.976139 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:34.542358 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:34.542325 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:42.024819 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.024790 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:42.117921 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.117885 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:42.199343 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.199313 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j98sb"] Jul 22 18:51:42.202542 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.202521 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j98sb" Jul 22 18:51:42.205322 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.205300 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 18:51:42.206083 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.206061 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 18:51:42.206207 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.206061 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-twj8b\"" Jul 22 18:51:42.213802 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.213781 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j98sb"] Jul 22 18:51:42.278702 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.278638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78mmb\" (UniqueName: \"kubernetes.io/projected/38f59f5f-c160-495c-bedf-65b1c284b7db-kube-api-access-78mmb\") pod \"limitador-operator-controller-manager-859d7b78bd-j98sb\" (UID: \"38f59f5f-c160-495c-bedf-65b1c284b7db\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j98sb" Jul 22 18:51:42.379191 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.379168 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78mmb\" (UniqueName: \"kubernetes.io/projected/38f59f5f-c160-495c-bedf-65b1c284b7db-kube-api-access-78mmb\") pod \"limitador-operator-controller-manager-859d7b78bd-j98sb\" (UID: \"38f59f5f-c160-495c-bedf-65b1c284b7db\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j98sb" Jul 22 18:51:42.390889 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.390861 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78mmb\" (UniqueName: \"kubernetes.io/projected/38f59f5f-c160-495c-bedf-65b1c284b7db-kube-api-access-78mmb\") pod \"limitador-operator-controller-manager-859d7b78bd-j98sb\" (UID: \"38f59f5f-c160-495c-bedf-65b1c284b7db\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j98sb" Jul 22 18:51:42.441822 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.441777 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:42.514892 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.514863 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j98sb" Jul 22 18:51:42.535732 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.535680 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:42.654443 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.654415 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j98sb"] Jul 22 18:51:42.656964 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:51:42.656938 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f59f5f_c160_495c_bedf_65b1c284b7db.slice/crio-1809ffee859af6e45f33ebccb96bb3df989047fbafa90b664d0158dff500dde9 WatchSource:0}: Error finding container 1809ffee859af6e45f33ebccb96bb3df989047fbafa90b664d0158dff500dde9: Status 404 returned error can't find the container with id 1809ffee859af6e45f33ebccb96bb3df989047fbafa90b664d0158dff500dde9 Jul 22 18:51:42.659382 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:42.659365 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 18:51:43.528381 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:43.528345 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j98sb" event={"ID":"38f59f5f-c160-495c-bedf-65b1c284b7db","Type":"ContainerStarted","Data":"1809ffee859af6e45f33ebccb96bb3df989047fbafa90b664d0158dff500dde9"} Jul 22 18:51:44.534385 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:44.534353 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j98sb" event={"ID":"38f59f5f-c160-495c-bedf-65b1c284b7db","Type":"ContainerStarted","Data":"da0fb2a70253ae8fb269aeb73445ceed94491642e3af3d7ae8f1765a2d977fcb"} Jul 22 18:51:44.534786 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:44.534459 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j98sb" Jul 22 18:51:44.551924 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:44.551861 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j98sb" podStartSLOduration=0.854917733 podStartE2EDuration="2.551846343s" totalImagesPullingTime="1.69692861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:51:44.550829236 +0000 UTC m=+714.998224470" watchObservedRunningTime="2026-07-22 18:51:44.551846343 +0000 UTC m=+714.999241575" Jul 22 18:51:45.614380 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:45.614346 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:45.714889 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:45.714860 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:45.826545 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:45.826516 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:45.929854 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:45.929785 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:46.128031 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:46.127998 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:46.255029 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:46.255000 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:46.763588 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:46.763556 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:47.280935 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:47.280901 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:47.334801 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:47.334771 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:52.579905 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:52.579859 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:52.678954 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:52.678916 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:55.540820 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:55.540790 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j98sb" Jul 22 18:51:57.276108 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:57.276068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:57.334756 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:57.334729 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:51:57.390172 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:51:57.390148 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:52:22.857954 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:52:22.857917 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:52:23.117805 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:52:23.117718 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:52:35.652994 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:52:35.652940 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:52:35.774716 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:52:35.774685 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:52:36.385962 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:52:36.385929 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:52:36.542043 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:52:36.542009 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:52:51.919883 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:52:51.919850 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:52:52.019627 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:52:52.019599 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:52:52.469831 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:52:52.469801 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:04.880287 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:04.880251 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:05.014629 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:05.014591 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:05.131859 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:05.131788 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4wcg2"] Jul 22 18:53:05.134991 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:05.134962 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4wcg2" Jul 22 18:53:05.137360 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:05.137340 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vt7cj\"" Jul 22 18:53:05.140847 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:05.140826 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4wcg2"] Jul 22 18:53:05.211882 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:05.211853 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67zj\" (UniqueName: \"kubernetes.io/projected/20e75e67-ef3e-4095-9ede-9aea56175961-kube-api-access-r67zj\") pod \"authorino-5d4b67c446-4wcg2\" (UID: \"20e75e67-ef3e-4095-9ede-9aea56175961\") " pod="kuadrant-system/authorino-5d4b67c446-4wcg2" Jul 22 18:53:05.312319 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:05.312291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67zj\" (UniqueName: \"kubernetes.io/projected/20e75e67-ef3e-4095-9ede-9aea56175961-kube-api-access-r67zj\") pod \"authorino-5d4b67c446-4wcg2\" (UID: \"20e75e67-ef3e-4095-9ede-9aea56175961\") " pod="kuadrant-system/authorino-5d4b67c446-4wcg2" Jul 22 18:53:05.312701 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:05.312683 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4wcg2"] Jul 22 18:53:05.312858 ip-10-0-143-15 kubenswrapper[2574]: E0722 18:53:05.312841 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-r67zj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-4wcg2" podUID="20e75e67-ef3e-4095-9ede-9aea56175961" Jul 22 18:53:05.325541 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:05.325517 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67zj\" (UniqueName: \"kubernetes.io/projected/20e75e67-ef3e-4095-9ede-9aea56175961-kube-api-access-r67zj\") pod \"authorino-5d4b67c446-4wcg2\" (UID: \"20e75e67-ef3e-4095-9ede-9aea56175961\") " pod="kuadrant-system/authorino-5d4b67c446-4wcg2" Jul 22 18:53:05.870081 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:05.870049 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4wcg2" Jul 22 18:53:05.876910 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:05.876887 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4wcg2" Jul 22 18:53:06.017343 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:06.017317 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r67zj\" (UniqueName: \"kubernetes.io/projected/20e75e67-ef3e-4095-9ede-9aea56175961-kube-api-access-r67zj\") pod \"20e75e67-ef3e-4095-9ede-9aea56175961\" (UID: \"20e75e67-ef3e-4095-9ede-9aea56175961\") " Jul 22 18:53:06.019317 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:06.019297 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e75e67-ef3e-4095-9ede-9aea56175961-kube-api-access-r67zj" (OuterVolumeSpecName: "kube-api-access-r67zj") pod "20e75e67-ef3e-4095-9ede-9aea56175961" (UID: "20e75e67-ef3e-4095-9ede-9aea56175961"). InnerVolumeSpecName "kube-api-access-r67zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 18:53:06.117933 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:06.117906 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r67zj\" (UniqueName: \"kubernetes.io/projected/20e75e67-ef3e-4095-9ede-9aea56175961-kube-api-access-r67zj\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 18:53:06.874590 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:06.874557 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4wcg2" Jul 22 18:53:06.903306 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:06.903273 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4wcg2"] Jul 22 18:53:06.905735 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:06.905709 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4wcg2"] Jul 22 18:53:08.004648 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:08.004613 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20e75e67-ef3e-4095-9ede-9aea56175961" path="/var/lib/kubelet/pods/20e75e67-ef3e-4095-9ede-9aea56175961/volumes" Jul 22 18:53:11.440986 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:11.440930 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:11.538624 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:11.538592 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:11.625176 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:11.625140 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:15.162436 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:15.162358 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:15.273012 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:15.272955 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:26.512892 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:26.512857 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:45.567099 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:45.567063 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:45.662416 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:45.662387 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:46.100657 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:46.100626 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:46.408746 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:46.408647 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:46.801685 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:46.801659 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:46.902230 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:46.902201 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:46.909466 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:46.909434 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-h4rb8"] Jul 22 18:53:46.912801 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:46.912783 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-h4rb8" Jul 22 18:53:46.915237 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:46.915219 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5kmmn\"" Jul 22 18:53:46.915328 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:46.915219 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 22 18:53:46.924223 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:46.924202 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-h4rb8"] Jul 22 18:53:46.950831 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:46.950808 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-h4rb8\" (UID: \"ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-h4rb8" Jul 22 18:53:46.950930 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:46.950836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfflh\" (UniqueName: \"kubernetes.io/projected/ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a-kube-api-access-pfflh\") pod \"payload-pre-processing-7677b774bb-h4rb8\" (UID: \"ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-h4rb8" Jul 22 18:53:47.051218 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:47.051193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-h4rb8\" (UID: \"ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-h4rb8" Jul 22 18:53:47.051355 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:47.051226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfflh\" (UniqueName: \"kubernetes.io/projected/ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a-kube-api-access-pfflh\") pod \"payload-pre-processing-7677b774bb-h4rb8\" (UID: \"ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-h4rb8" Jul 22 18:53:47.051881 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:47.051831 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-h4rb8\" (UID: \"ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-h4rb8" Jul 22 18:53:47.059197 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:47.059179 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfflh\" (UniqueName: \"kubernetes.io/projected/ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a-kube-api-access-pfflh\") pod \"payload-pre-processing-7677b774bb-h4rb8\" (UID: \"ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-h4rb8" Jul 22 18:53:47.223337 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:47.223304 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-h4rb8" Jul 22 18:53:47.357162 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:47.357140 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-h4rb8"] Jul 22 18:53:47.359840 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:53:47.359811 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7f5c1b_1c3b_4a9d_9514_b6616aa94b2a.slice/crio-72f39c210cc2ea929572435c0be060012f86730429845c87a6bd7184b62b22e1 WatchSource:0}: Error finding container 72f39c210cc2ea929572435c0be060012f86730429845c87a6bd7184b62b22e1: Status 404 returned error can't find the container with id 72f39c210cc2ea929572435c0be060012f86730429845c87a6bd7184b62b22e1 Jul 22 18:53:48.054921 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:48.054887 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-h4rb8" event={"ID":"ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a","Type":"ContainerStarted","Data":"72f39c210cc2ea929572435c0be060012f86730429845c87a6bd7184b62b22e1"} Jul 22 18:53:49.060718 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:49.060685 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-h4rb8" event={"ID":"ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a","Type":"ContainerStarted","Data":"e20cbe2ce79ed00d8b28ff85fcc5d5ac2bd60417dbd7a2edf5cd4970a4f0d5e7"} Jul 22 18:53:49.061092 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:49.060867 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-h4rb8" Jul 22 18:53:49.074519 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:49.074469 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-h4rb8" podStartSLOduration=1.5542138250000002 podStartE2EDuration="3.074447687s" totalImagesPullingTime="1.520233862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:53:49.074251995 +0000 UTC m=+839.521647237" watchObservedRunningTime="2026-07-22 18:53:49.074447687 +0000 UTC m=+839.521842911" Jul 22 18:53:56.833196 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:56.833161 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:53:56.945899 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:53:56.945865 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:54:00.067103 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:00.067072 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-h4rb8" Jul 22 18:54:02.317564 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:02.317529 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:54:02.467559 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:02.467531 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:54:04.486722 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.486687 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8"] Jul 22 18:54:04.492253 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.492234 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.494813 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.494792 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 18:54:04.494912 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.494852 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-92cn2\"" Jul 22 18:54:04.495533 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.495516 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 22 18:54:04.495585 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.495546 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 18:54:04.502928 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.502907 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8"] Jul 22 18:54:04.523808 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.523779 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:54:04.595229 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.595197 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.595229 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.595230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.595439 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.595247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.595439 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.595337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c45rt\" (UniqueName: \"kubernetes.io/projected/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-kube-api-access-c45rt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.595439 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.595373 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.595439 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.595393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.695953 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.695902 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c45rt\" (UniqueName: \"kubernetes.io/projected/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-kube-api-access-c45rt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.696143 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.695989 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.696143 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.696017 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.696143 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.696054 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.696143 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.696079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.696143 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.696099 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.696476 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.696453 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.696547 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.696506 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.696547 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.696536 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.698351 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.698329 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.698478 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.698461 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.704355 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.704333 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c45rt\" (UniqueName: \"kubernetes.io/projected/76c042e5-202b-4e4f-8fa3-ed9e18fffc9f-kube-api-access-c45rt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8\" (UID: \"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.802200 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.802139 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:04.929909 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:04.929876 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8"] Jul 22 18:54:04.933739 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:54:04.933712 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c042e5_202b_4e4f_8fa3_ed9e18fffc9f.slice/crio-2cdb153e0ca19b20d5d4f7c850a63a0e5aed8b7c31c4da1566e2af7b141fda2c WatchSource:0}: Error finding container 2cdb153e0ca19b20d5d4f7c850a63a0e5aed8b7c31c4da1566e2af7b141fda2c: Status 404 returned error can't find the container with id 2cdb153e0ca19b20d5d4f7c850a63a0e5aed8b7c31c4da1566e2af7b141fda2c Jul 22 18:54:05.126342 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:05.126274 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" event={"ID":"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f","Type":"ContainerStarted","Data":"2cdb153e0ca19b20d5d4f7c850a63a0e5aed8b7c31c4da1566e2af7b141fda2c"} Jul 22 18:54:07.138863 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:07.138826 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" event={"ID":"76c042e5-202b-4e4f-8fa3-ed9e18fffc9f","Type":"ContainerStarted","Data":"21a015b29c6bb83704daffbb6e0bb24d9a74377b77481add0fc8e958efe378cf"} Jul 22 18:54:07.156474 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:07.156427 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" podStartSLOduration=1.992447834 podStartE2EDuration="3.15641112s" totalImagesPullingTime="1.163963286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:54:07.155487028 +0000 UTC m=+857.602882260" watchObservedRunningTime="2026-07-22 18:54:07.15641112 +0000 UTC m=+857.603806403" Jul 22 18:54:11.821214 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:11.821177 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:54:14.803209 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:14.803167 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:14.803582 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:14.803235 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:14.815375 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:14.815351 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:15.188661 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:15.188591 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8" Jul 22 18:54:22.212427 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:22.212393 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:54:44.216570 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:44.216484 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:54:56.491450 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.491417 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd"] Jul 22 18:54:56.494692 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.494676 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.497141 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.497118 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 22 18:54:56.505657 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.505633 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd"] Jul 22 18:54:56.522025 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.522004 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:54:56.605156 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.605125 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10818ee-3735-4efb-ba1f-5ea97839b358-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.605286 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.605163 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10818ee-3735-4efb-ba1f-5ea97839b358-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.605286 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.605182 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10818ee-3735-4efb-ba1f-5ea97839b358-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.605286 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.605253 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10818ee-3735-4efb-ba1f-5ea97839b358-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.605388 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.605305 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10818ee-3735-4efb-ba1f-5ea97839b358-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.605388 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.605322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mgwz\" (UniqueName: \"kubernetes.io/projected/f10818ee-3735-4efb-ba1f-5ea97839b358-kube-api-access-5mgwz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.706551 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.706524 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10818ee-3735-4efb-ba1f-5ea97839b358-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.706680 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.706553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mgwz\" (UniqueName: \"kubernetes.io/projected/f10818ee-3735-4efb-ba1f-5ea97839b358-kube-api-access-5mgwz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.706680 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.706583 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10818ee-3735-4efb-ba1f-5ea97839b358-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.706750 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.706700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10818ee-3735-4efb-ba1f-5ea97839b358-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.706750 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.706744 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10818ee-3735-4efb-ba1f-5ea97839b358-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.706819 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.706797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10818ee-3735-4efb-ba1f-5ea97839b358-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.707002 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.706959 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10818ee-3735-4efb-ba1f-5ea97839b358-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.707053 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.707010 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10818ee-3735-4efb-ba1f-5ea97839b358-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.707141 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.707125 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10818ee-3735-4efb-ba1f-5ea97839b358-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.708842 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.708823 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10818ee-3735-4efb-ba1f-5ea97839b358-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.709182 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.709165 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10818ee-3735-4efb-ba1f-5ea97839b358-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.716761 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.716736 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mgwz\" (UniqueName: \"kubernetes.io/projected/f10818ee-3735-4efb-ba1f-5ea97839b358-kube-api-access-5mgwz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lpzhd\" (UID: \"f10818ee-3735-4efb-ba1f-5ea97839b358\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.804556 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.804500 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:54:56.934768 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:56.934738 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd"] Jul 22 18:54:56.938629 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:54:56.938592 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10818ee_3735_4efb_ba1f_5ea97839b358.slice/crio-3eed1285582a9d01ea31e64f41ab1e7b89b50098fb8b02e1f199772e677efb8a WatchSource:0}: Error finding container 3eed1285582a9d01ea31e64f41ab1e7b89b50098fb8b02e1f199772e677efb8a: Status 404 returned error can't find the container with id 3eed1285582a9d01ea31e64f41ab1e7b89b50098fb8b02e1f199772e677efb8a Jul 22 18:54:57.355708 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:57.355673 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" event={"ID":"f10818ee-3735-4efb-ba1f-5ea97839b358","Type":"ContainerStarted","Data":"8705fa9a780db0602f079ab86fddc40fffab307fe9a940d0af1bdf28e28149bb"} Jul 22 18:54:57.355877 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:57.355722 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" event={"ID":"f10818ee-3735-4efb-ba1f-5ea97839b358","Type":"ContainerStarted","Data":"3eed1285582a9d01ea31e64f41ab1e7b89b50098fb8b02e1f199772e677efb8a"} Jul 22 18:54:57.375446 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:54:57.375400 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" podStartSLOduration=1.136603481 podStartE2EDuration="1.375386552s" totalImagesPullingTime="238.783071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:54:57.374894816 +0000 UTC m=+907.822290052" watchObservedRunningTime="2026-07-22 18:54:57.375386552 +0000 UTC m=+907.822781784" Jul 22 18:55:06.805107 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:06.805060 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:55:06.805581 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:06.805127 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:55:06.817417 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:06.817393 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:55:07.409280 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:07.409248 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lpzhd" Jul 22 18:55:07.981376 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:07.981341 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6"] Jul 22 18:55:07.985380 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:07.985358 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:07.987670 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:07.987652 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 22 18:55:07.993906 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:07.993884 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6"] Jul 22 18:55:08.009434 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.009411 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:55:08.097479 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.097458 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427fa00a-700d-4abb-ac22-037f50135707-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.097579 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.097498 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427fa00a-700d-4abb-ac22-037f50135707-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.097713 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.097693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlpqc\" (UniqueName: \"kubernetes.io/projected/427fa00a-700d-4abb-ac22-037f50135707-kube-api-access-rlpqc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.097771 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.097755 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427fa00a-700d-4abb-ac22-037f50135707-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.097937 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.097913 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427fa00a-700d-4abb-ac22-037f50135707-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.098054 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.097992 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427fa00a-700d-4abb-ac22-037f50135707-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.198877 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.198849 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427fa00a-700d-4abb-ac22-037f50135707-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.199031 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.198889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlpqc\" (UniqueName: \"kubernetes.io/projected/427fa00a-700d-4abb-ac22-037f50135707-kube-api-access-rlpqc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.199031 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.198914 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427fa00a-700d-4abb-ac22-037f50135707-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.199031 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.198942 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427fa00a-700d-4abb-ac22-037f50135707-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.199031 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.198961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427fa00a-700d-4abb-ac22-037f50135707-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.199031 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.199019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427fa00a-700d-4abb-ac22-037f50135707-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.199350 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.199309 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427fa00a-700d-4abb-ac22-037f50135707-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.199406 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.199354 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427fa00a-700d-4abb-ac22-037f50135707-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.199472 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.199455 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427fa00a-700d-4abb-ac22-037f50135707-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.200941 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.200926 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427fa00a-700d-4abb-ac22-037f50135707-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.201319 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.201304 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427fa00a-700d-4abb-ac22-037f50135707-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.206447 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.206428 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlpqc\" (UniqueName: \"kubernetes.io/projected/427fa00a-700d-4abb-ac22-037f50135707-kube-api-access-rlpqc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6\" (UID: \"427fa00a-700d-4abb-ac22-037f50135707\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.295477 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.295417 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:08.425704 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:08.425676 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6"] Jul 22 18:55:08.427206 ip-10-0-143-15 kubenswrapper[2574]: W0722 18:55:08.427184 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod427fa00a_700d_4abb_ac22_037f50135707.slice/crio-2838d2718683f3d2096543f7ca98dff0eba6ae5a2a7d0c9ef6b14ad171a44da4 WatchSource:0}: Error finding container 2838d2718683f3d2096543f7ca98dff0eba6ae5a2a7d0c9ef6b14ad171a44da4: Status 404 returned error can't find the container with id 2838d2718683f3d2096543f7ca98dff0eba6ae5a2a7d0c9ef6b14ad171a44da4 Jul 22 18:55:09.410614 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:09.410581 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" event={"ID":"427fa00a-700d-4abb-ac22-037f50135707","Type":"ContainerStarted","Data":"452c5a8dd8146a5ab712298c8ae5069056d15807a3c3ace97b153980ea421c80"} Jul 22 18:55:09.410614 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:09.410621 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" event={"ID":"427fa00a-700d-4abb-ac22-037f50135707","Type":"ContainerStarted","Data":"2838d2718683f3d2096543f7ca98dff0eba6ae5a2a7d0c9ef6b14ad171a44da4"} Jul 22 18:55:09.427955 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:09.427898 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" podStartSLOduration=2.0228472220000002 podStartE2EDuration="2.427884279s" totalImagesPullingTime="405.037057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:55:09.427387843 +0000 UTC m=+919.874783075" watchObservedRunningTime="2026-07-22 18:55:09.427884279 +0000 UTC m=+919.875279513" Jul 22 18:55:18.295696 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:18.295657 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:18.295696 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:18.295705 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:18.308090 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:18.308070 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:18.465279 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:18.465245 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6" Jul 22 18:55:57.143853 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:57.143821 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:55:59.590151 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:55:59.590106 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:56:04.920294 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:56:04.920256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:56:04.972927 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:56:04.972896 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:56:14.575382 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:56:14.575297 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:56:14.719741 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:56:14.719706 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:56:15.782911 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:56:15.782874 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:56:15.885108 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:56:15.885073 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:56:59.430225 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:56:59.430190 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:57:45.234313 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:57:45.234226 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:58:20.590308 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:58:20.590272 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:58:21.775106 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:58:21.775072 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:58:34.670998 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:58:34.670940 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:58:34.770833 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:58:34.770791 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:58:57.910387 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:58:57.910354 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:58:58.012134 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:58:58.012093 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:58:58.152666 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:58:58.152630 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:59:04.046656 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:59:04.046616 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:59:04.151093 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:59:04.151059 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:59:04.253334 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:59:04.253301 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:59:04.444944 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:59:04.444851 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:59:10.133518 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:59:10.133482 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:59:10.198470 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:59:10.198434 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:59:52.473571 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:59:52.473535 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 18:59:52.527963 ip-10-0-143-15 kubenswrapper[2574]: I0722 18:59:52.527932 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:02:00.528937 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:02:00.528896 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:02:14.360822 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:02:14.360741 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:02:14.410722 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:02:14.410693 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:02:31.944645 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:02:31.944600 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:04:25.010500 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:04:25.010417 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:06:55.840919 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:06:55.840837 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:07:01.193400 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.193359 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s"] Jul 22 19:07:01.196831 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.196809 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.198995 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.198958 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-df776eca-kserve-self-signed-certs\"" Jul 22 19:07:01.205797 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.205775 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s"] Jul 22 19:07:01.293181 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.293153 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:07:01.318510 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.318483 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-tmp-dir\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.318579 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.318564 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-dshm\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.318634 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.318620 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-home\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.318694 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.318679 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-tls-certs\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.318743 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.318711 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgtxn\" (UniqueName: \"kubernetes.io/projected/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-kube-api-access-dgtxn\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.318787 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.318774 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-model-cache\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.419879 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.419840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-dshm\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.420074 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.419893 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-home\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.420074 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.419926 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-tls-certs\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.420074 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.419945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgtxn\" (UniqueName: \"kubernetes.io/projected/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-kube-api-access-dgtxn\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.420074 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.420011 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-model-cache\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.420293 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.420142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-tmp-dir\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.420413 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.420388 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-home\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.420550 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.420400 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-model-cache\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.420550 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.420454 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-tmp-dir\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.422695 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.422678 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-dshm\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.422952 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.422935 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-tls-certs\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.427052 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.427028 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgtxn\" (UniqueName: \"kubernetes.io/projected/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-kube-api-access-dgtxn\") pod \"e2e-same-id-b-df776eca-kserve-9499f9495-cm72s\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.506330 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.506302 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:01.635527 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:01.635500 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s"] Jul 22 19:07:01.637914 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:07:01.637887 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0b117fb_4d10_4a3a_b343_92cfef21ec3d.slice/crio-310796b4e41d7e2a2b8f6e68f4e50f9f149056f4a701b23dde480c74425fbdfb WatchSource:0}: Error finding container 310796b4e41d7e2a2b8f6e68f4e50f9f149056f4a701b23dde480c74425fbdfb: Status 404 returned error can't find the container with id 310796b4e41d7e2a2b8f6e68f4e50f9f149056f4a701b23dde480c74425fbdfb Jul 22 19:07:02.580779 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:02.580744 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" event={"ID":"d0b117fb-4d10-4a3a-b343-92cfef21ec3d","Type":"ContainerStarted","Data":"60b536c9b264bb08c86c05d4395763bd726957cc468e447ad91a7f29c0264348"} Jul 22 19:07:02.581211 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:02.580786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" event={"ID":"d0b117fb-4d10-4a3a-b343-92cfef21ec3d","Type":"ContainerStarted","Data":"310796b4e41d7e2a2b8f6e68f4e50f9f149056f4a701b23dde480c74425fbdfb"} Jul 22 19:07:02.599227 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:02.599178 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" podStartSLOduration=1.5991635199999998 podStartE2EDuration="1.59916352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:07:02.596722815 +0000 UTC m=+1633.044118048" watchObservedRunningTime="2026-07-22 19:07:02.59916352 +0000 UTC m=+1633.046558751" Jul 22 19:07:11.507150 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:11.507111 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:11.507557 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:11.507186 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:11.519352 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:11.519329 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:11.631846 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:11.631816 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:07:45.945292 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:45.945256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:07:47.218608 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:47.218576 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s"] Jul 22 19:07:47.413444 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:07:47.413412 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:08:02.238342 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.238302 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" podUID="d0b117fb-4d10-4a3a-b343-92cfef21ec3d" containerName="main" containerID="cri-o://60b536c9b264bb08c86c05d4395763bd726957cc468e447ad91a7f29c0264348" gracePeriod=45 Jul 22 19:08:02.481017 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.480990 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:08:02.645171 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.645103 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-tmp-dir\") pod \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " Jul 22 19:08:02.645171 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.645138 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgtxn\" (UniqueName: \"kubernetes.io/projected/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-kube-api-access-dgtxn\") pod \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " Jul 22 19:08:02.645330 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.645218 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-home\") pod \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " Jul 22 19:08:02.645330 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.645255 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-model-cache\") pod \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " Jul 22 19:08:02.645330 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.645298 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-tls-certs\") pod \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " Jul 22 19:08:02.645480 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.645329 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-dshm\") pod \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\" (UID: \"d0b117fb-4d10-4a3a-b343-92cfef21ec3d\") " Jul 22 19:08:02.645480 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.645372 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0b117fb-4d10-4a3a-b343-92cfef21ec3d" (UID: "d0b117fb-4d10-4a3a-b343-92cfef21ec3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:08:02.645582 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.645532 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-home" (OuterVolumeSpecName: "home") pod "d0b117fb-4d10-4a3a-b343-92cfef21ec3d" (UID: "d0b117fb-4d10-4a3a-b343-92cfef21ec3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:08:02.645582 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.645547 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0b117fb-4d10-4a3a-b343-92cfef21ec3d" (UID: "d0b117fb-4d10-4a3a-b343-92cfef21ec3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:08:02.645690 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.645585 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:08:02.647436 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.647409 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-kube-api-access-dgtxn" (OuterVolumeSpecName: "kube-api-access-dgtxn") pod "d0b117fb-4d10-4a3a-b343-92cfef21ec3d" (UID: "d0b117fb-4d10-4a3a-b343-92cfef21ec3d"). InnerVolumeSpecName "kube-api-access-dgtxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:08:02.647546 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.647444 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0b117fb-4d10-4a3a-b343-92cfef21ec3d" (UID: "d0b117fb-4d10-4a3a-b343-92cfef21ec3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:08:02.647546 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.647487 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-dshm" (OuterVolumeSpecName: "dshm") pod "d0b117fb-4d10-4a3a-b343-92cfef21ec3d" (UID: "d0b117fb-4d10-4a3a-b343-92cfef21ec3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:08:02.747023 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.746999 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:08:02.747023 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.747022 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:08:02.747177 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.747035 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:08:02.747177 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.747047 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgtxn\" (UniqueName: \"kubernetes.io/projected/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-kube-api-access-dgtxn\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:08:02.747177 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.747056 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b117fb-4d10-4a3a-b343-92cfef21ec3d-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:08:02.853240 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.853208 2574 generic.go:356] "Generic (PLEG): container finished" podID="d0b117fb-4d10-4a3a-b343-92cfef21ec3d" containerID="60b536c9b264bb08c86c05d4395763bd726957cc468e447ad91a7f29c0264348" exitCode=0 Jul 22 19:08:02.853394 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.853255 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" Jul 22 19:08:02.853394 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.853293 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" event={"ID":"d0b117fb-4d10-4a3a-b343-92cfef21ec3d","Type":"ContainerDied","Data":"60b536c9b264bb08c86c05d4395763bd726957cc468e447ad91a7f29c0264348"} Jul 22 19:08:02.853394 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.853336 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s" event={"ID":"d0b117fb-4d10-4a3a-b343-92cfef21ec3d","Type":"ContainerDied","Data":"310796b4e41d7e2a2b8f6e68f4e50f9f149056f4a701b23dde480c74425fbdfb"} Jul 22 19:08:02.853394 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.853353 2574 scope.go:117] "RemoveContainer" containerID="60b536c9b264bb08c86c05d4395763bd726957cc468e447ad91a7f29c0264348" Jul 22 19:08:02.862996 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.862956 2574 scope.go:117] "RemoveContainer" containerID="60b536c9b264bb08c86c05d4395763bd726957cc468e447ad91a7f29c0264348" Jul 22 19:08:02.863292 ip-10-0-143-15 kubenswrapper[2574]: E0722 19:08:02.863264 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b536c9b264bb08c86c05d4395763bd726957cc468e447ad91a7f29c0264348\": container with ID starting with 60b536c9b264bb08c86c05d4395763bd726957cc468e447ad91a7f29c0264348 not found: ID does not exist" containerID="60b536c9b264bb08c86c05d4395763bd726957cc468e447ad91a7f29c0264348" Jul 22 19:08:02.863341 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.863299 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b536c9b264bb08c86c05d4395763bd726957cc468e447ad91a7f29c0264348"} err="failed to get container status \"60b536c9b264bb08c86c05d4395763bd726957cc468e447ad91a7f29c0264348\": rpc error: code = NotFound desc = could not find container \"60b536c9b264bb08c86c05d4395763bd726957cc468e447ad91a7f29c0264348\": container with ID starting with 60b536c9b264bb08c86c05d4395763bd726957cc468e447ad91a7f29c0264348 not found: ID does not exist" Jul 22 19:08:02.881986 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.881950 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s"] Jul 22 19:08:02.889983 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:02.889954 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-df776eca-kserve-9499f9495-cm72s"] Jul 22 19:08:04.005784 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:08:04.005742 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0b117fb-4d10-4a3a-b343-92cfef21ec3d" path="/var/lib/kubelet/pods/d0b117fb-4d10-4a3a-b343-92cfef21ec3d/volumes" Jul 22 19:10:01.221335 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:10:01.221296 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:10:50.272633 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:10:50.272599 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:10:50.328980 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:10:50.328932 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:02.776492 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:02.776457 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:26.414700 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:26.414602 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:29.640433 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:29.640396 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:29.746587 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:29.746550 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:30.078679 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:30.078643 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:35.647476 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:35.647432 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:35.721499 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:35.721463 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:35.897106 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:35.897073 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:36.229609 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:36.229576 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:36.371179 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:36.371141 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:38.878291 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:38.878254 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:39.014189 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:39.014153 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:39.127452 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:39.127411 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:39.564834 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:39.564800 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:39.671756 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:39.671727 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:11:39.765771 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:11:39.765742 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:15.342594 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:15.342560 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:19.586592 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:19.586559 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:19.753028 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:19.752961 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:20.004274 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:20.004245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:20.345411 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:20.345320 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:20.456391 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:20.456360 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:20.556003 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:20.555951 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:20.881495 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:20.881462 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:21.100549 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:21.100517 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:23.213151 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:23.213109 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:23.328401 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:23.328360 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:23.748787 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:23.748751 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:23.850092 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:23.850061 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:12:23.950330 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:12:23.950300 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:08.827466 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:08.827423 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:13.345578 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:13.345542 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:13.534528 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:13.534491 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:13.766213 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:13.766185 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:14.010789 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:14.010762 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:14.341081 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:14.341042 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:14.580893 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:14.580862 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:16.424044 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:16.424008 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:16.523187 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:16.523146 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:16.718436 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:16.718359 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:16.925726 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:16.925687 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:17.197501 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:17.197467 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:17.298199 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:17.298161 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:43.741323 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:43.741276 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:43.838979 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:43.838947 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:52.936232 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:52.936190 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:53.037124 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:53.037082 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:53.171027 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:53.170990 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:57.415880 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:57.415845 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:58.783195 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:58.783164 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:58.969244 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:58.969210 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:59.292899 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:59.292866 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:59.579558 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:59.579465 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:13:59.763355 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:13:59.763319 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:01.662907 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:01.662849 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:01.758862 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:01.758833 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:01.859532 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:01.859496 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:02.037856 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:02.037827 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:02.319951 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:02.319860 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:02.416759 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:02.416717 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:37.114135 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:37.114081 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:42.425917 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:42.425876 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:42.482861 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:42.482827 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:42.860550 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:42.860521 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:43.769076 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:43.769045 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:44.541936 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:44.541903 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:44.642727 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:44.642695 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:45.864869 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:45.864834 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:52.080496 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:52.080450 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:52.188206 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:52.188178 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:52.286161 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:52.286117 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:52.381825 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:52.381733 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:14:52.894621 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:14:52.894582 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:15:18.920907 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:15:18.920870 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:15:26.411845 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:15:26.411790 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:15:26.542388 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:15:26.542352 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:15:41.568298 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:15:41.568187 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:15:47.297023 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:15:47.296986 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:15:47.405201 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:15:47.405148 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:15:47.501126 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:15:47.501092 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:15:58.099294 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:15:58.099255 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:15:58.930652 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:15:58.930621 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:05.160426 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:05.160387 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:05.299621 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:05.299592 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:05.389421 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:05.389380 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:05.485201 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:05.485167 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:06.272964 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:06.272926 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:06.372433 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:06.372402 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:06.473265 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:06.473209 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:08.171336 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:08.171295 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:13.958301 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:13.958259 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:14.061668 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:14.061628 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:14.658814 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:14.658777 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:20.493452 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:20.493413 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:20.560242 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:20.560211 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:20.735722 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:20.735688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:21.063491 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:21.063449 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:21.528923 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:21.528890 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:22.149440 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:22.149406 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:22.212563 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:22.212531 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:24.064012 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:24.063951 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:24.160829 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:24.160798 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:24.332850 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:24.332756 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:24.605217 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:24.605135 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:24.936169 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:24.936077 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:25.036587 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:25.036553 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:28.612955 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:28.612912 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:29.301579 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:29.301545 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:30.126527 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:30.126489 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:30.230769 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:30.230740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:35.594100 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:35.594066 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:35.657680 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:35.657645 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:46.389824 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:46.389781 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:51.034781 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:51.034742 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:51.438452 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:51.438340 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:57.234801 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:57.234761 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:57.336874 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:57.336844 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:57.613568 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:57.613480 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:57.785261 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:57.785229 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:57.931930 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:57.931836 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:58.024153 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:58.024110 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:58.140747 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:58.140716 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:16:59.774892 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:16:59.774852 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:04.088469 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:04.088425 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:04.149853 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:04.149822 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:04.240589 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:04.240556 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:04.339940 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:04.339862 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:04.456859 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:04.456827 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:10.296408 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:10.296370 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:10.374905 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:10.374870 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:17.247383 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:17.247350 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:17.591081 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:17.590988 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:17.722099 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:17.722067 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:20.420116 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:20.420075 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:21.362200 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:21.362161 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:21.473826 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:21.473793 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:24.924725 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:24.924686 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:17:30.879068 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:17:30.879033 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:18:03.603086 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:03.603050 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h"] Jul 22 19:18:03.603549 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:03.603390 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b117fb-4d10-4a3a-b343-92cfef21ec3d" containerName="main" Jul 22 19:18:03.603549 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:03.603401 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b117fb-4d10-4a3a-b343-92cfef21ec3d" containerName="main" Jul 22 19:18:03.603549 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:03.603468 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0b117fb-4d10-4a3a-b343-92cfef21ec3d" containerName="main" Jul 22 19:18:03.606648 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:03.606631 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h" Jul 22 19:18:03.608764 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:03.608744 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 19:18:03.608862 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:03.608849 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r5ttk\"" Jul 22 19:18:03.612758 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:03.612737 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h"] Jul 22 19:18:03.660949 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:03.660918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4abd57-1cac-49b5-b589-08a5d6c8a0cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-73899c00-m988h\" (UID: \"8a4abd57-1cac-49b5-b589-08a5d6c8a0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h" Jul 22 19:18:03.761283 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:03.761222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4abd57-1cac-49b5-b589-08a5d6c8a0cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-73899c00-m988h\" (UID: \"8a4abd57-1cac-49b5-b589-08a5d6c8a0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h" Jul 22 19:18:03.761793 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:03.761774 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4abd57-1cac-49b5-b589-08a5d6c8a0cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-73899c00-m988h\" (UID: \"8a4abd57-1cac-49b5-b589-08a5d6c8a0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h" Jul 22 19:18:03.918999 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:03.918923 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h" Jul 22 19:18:04.048039 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:04.048011 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h"] Jul 22 19:18:04.048929 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:18:04.048904 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4abd57_1cac_49b5_b589_08a5d6c8a0cd.slice/crio-4e48fc9b9be43100ae5d595fd46564d2086300460b6f24cb39ad691ddcb9ccd2 WatchSource:0}: Error finding container 4e48fc9b9be43100ae5d595fd46564d2086300460b6f24cb39ad691ddcb9ccd2: Status 404 returned error can't find the container with id 4e48fc9b9be43100ae5d595fd46564d2086300460b6f24cb39ad691ddcb9ccd2 Jul 22 19:18:04.050803 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:04.050783 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 19:18:04.508308 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:04.508273 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h" event={"ID":"8a4abd57-1cac-49b5-b589-08a5d6c8a0cd","Type":"ContainerStarted","Data":"4e48fc9b9be43100ae5d595fd46564d2086300460b6f24cb39ad691ddcb9ccd2"} Jul 22 19:18:18.571398 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:18.571362 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a4abd57-1cac-49b5-b589-08a5d6c8a0cd" containerID="12dbfbb097fb19f1d68e22decdaa33f61a466cc2bf405f667670b9ba5790c0ec" exitCode=0 Jul 22 19:18:18.571762 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:18.571442 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h" event={"ID":"8a4abd57-1cac-49b5-b589-08a5d6c8a0cd","Type":"ContainerDied","Data":"12dbfbb097fb19f1d68e22decdaa33f61a466cc2bf405f667670b9ba5790c0ec"} Jul 22 19:18:19.704249 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:19.704225 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h" Jul 22 19:18:19.801988 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:19.801938 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4abd57-1cac-49b5-b589-08a5d6c8a0cd-maas-api-service-ca\") pod \"8a4abd57-1cac-49b5-b589-08a5d6c8a0cd\" (UID: \"8a4abd57-1cac-49b5-b589-08a5d6c8a0cd\") " Jul 22 19:18:19.802282 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:19.802259 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a4abd57-1cac-49b5-b589-08a5d6c8a0cd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a4abd57-1cac-49b5-b589-08a5d6c8a0cd" (UID: "8a4abd57-1cac-49b5-b589-08a5d6c8a0cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:18:19.902873 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:19.902791 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4abd57-1cac-49b5-b589-08a5d6c8a0cd-maas-api-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:18:20.581391 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:20.581359 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h" Jul 22 19:18:20.581562 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:20.581389 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h" event={"ID":"8a4abd57-1cac-49b5-b589-08a5d6c8a0cd","Type":"ContainerDied","Data":"4e48fc9b9be43100ae5d595fd46564d2086300460b6f24cb39ad691ddcb9ccd2"} Jul 22 19:18:20.581562 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:20.581430 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e48fc9b9be43100ae5d595fd46564d2086300460b6f24cb39ad691ddcb9ccd2" Jul 22 19:18:29.040803 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:29.040768 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:18:33.681390 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:33.681357 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h"] Jul 22 19:18:33.684020 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:33.683993 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73899c00-m988h"] Jul 22 19:18:33.685948 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:33.685917 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:18:34.005513 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:34.005484 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a4abd57-1cac-49b5-b589-08a5d6c8a0cd" path="/var/lib/kubelet/pods/8a4abd57-1cac-49b5-b589-08a5d6c8a0cd/volumes" Jul 22 19:18:36.042291 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:36.042255 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:18:41.116641 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:41.116599 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:18:44.709073 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:44.708964 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:18:44.809243 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:44.809206 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:18:58.039901 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:18:58.039868 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:17.865750 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:17.865716 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:18.045243 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:18.045208 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:23.576239 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:23.576202 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:23.643330 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:23.643291 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:23.878156 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:23.878071 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:24.109451 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:24.109423 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:25.282455 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:25.282424 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:25.378250 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:25.378179 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:26.241453 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:26.241420 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:26.332948 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:26.332916 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:26.433389 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:26.433358 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:27.870141 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:27.870101 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:33.622776 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:33.622739 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:33.685927 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:33.685896 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:34.691684 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:34.691650 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:40.759577 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:40.759538 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:40.869252 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:40.869220 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:40.964533 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:40.964496 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:51.567134 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:51.567095 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:19:51.929533 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:19:51.929435 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:12.944684 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:12.944596 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:19.016109 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:19.016073 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:19.115297 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:19.115254 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:19.378922 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:19.378832 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:19.812511 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:19.812476 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:19.902886 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:19.902843 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:22.447022 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:22.446961 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:22.538693 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:22.538652 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:22.596765 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:22.596726 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:23.102965 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:23.102920 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:23.203767 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:23.203728 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:23.564208 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:23.564169 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:26.956117 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:26.956075 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:27.312183 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:27.312140 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:27.666443 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:27.666360 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:27.853056 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:27.853023 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:30.442076 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:30.442030 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:30.534823 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:30.534791 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:30.615527 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:30.615493 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:30.756278 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:30.756249 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:31.093477 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:31.093390 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:31.194131 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:31.194103 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:31.293343 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:31.293319 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:34.950026 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:34.949993 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:42.717733 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:42.717695 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:43.996897 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:43.996863 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:44.098354 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:44.098311 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:52.830436 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:52.830401 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:52.931790 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:52.931758 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:20:53.561256 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:20:53.561219 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:21:04.985101 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:04.985062 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh"] Jul 22 19:21:04.985484 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:04.985468 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a4abd57-1cac-49b5-b589-08a5d6c8a0cd" containerName="revoke-keys" Jul 22 19:21:04.985534 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:04.985487 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a4abd57-1cac-49b5-b589-08a5d6c8a0cd" containerName="revoke-keys" Jul 22 19:21:04.985572 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:04.985547 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a4abd57-1cac-49b5-b589-08a5d6c8a0cd" containerName="revoke-keys" Jul 22 19:21:04.988710 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:04.988689 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:04.992272 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:04.992250 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ba9716\"/\"default-dockercfg-sp6hr\"" Jul 22 19:21:04.992410 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:04.992266 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ba9716\"/\"test-model-ba9716-kserve-self-signed-certs\"" Jul 22 19:21:04.992410 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:04.992291 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ba9716\"/\"kube-root-ca.crt\"" Jul 22 19:21:04.992410 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:04.992345 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ba9716\"/\"openshift-service-ca.crt\"" Jul 22 19:21:05.000305 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.000285 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh"] Jul 22 19:21:05.018595 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.018567 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:21:05.068526 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.068503 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-home\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.068638 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.068539 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dfd9\" (UniqueName: \"kubernetes.io/projected/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-kube-api-access-9dfd9\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.068638 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.068571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-model-cache\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.068638 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.068624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-tmp-dir\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.068739 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.068678 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-tls-certs\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.068739 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.068694 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-dshm\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.169156 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.169132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-home\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.169276 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.169172 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfd9\" (UniqueName: \"kubernetes.io/projected/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-kube-api-access-9dfd9\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.169276 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.169208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-model-cache\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.169276 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.169248 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-tmp-dir\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.169461 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.169286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-tls-certs\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.169461 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.169311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-dshm\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.169558 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.169541 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-home\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.169621 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.169604 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-tmp-dir\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.169668 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.169647 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-model-cache\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.171543 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.171517 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-dshm\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.171771 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.171754 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-tls-certs\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.178042 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.178018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dfd9\" (UniqueName: \"kubernetes.io/projected/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-kube-api-access-9dfd9\") pod \"test-model-ba9716-kserve-77845bcc8f-pcnxh\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.298331 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.298254 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:05.430369 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:05.430344 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh"] Jul 22 19:21:05.432433 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:21:05.432408 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1983cd9f_91c7_4679_ac2a_a4e51ae8f8ff.slice/crio-ff0c3aa49908f9c198fdbba7c154a072d3434e9f37fcdd90441cbf3d07968cd3 WatchSource:0}: Error finding container ff0c3aa49908f9c198fdbba7c154a072d3434e9f37fcdd90441cbf3d07968cd3: Status 404 returned error can't find the container with id ff0c3aa49908f9c198fdbba7c154a072d3434e9f37fcdd90441cbf3d07968cd3 Jul 22 19:21:06.315119 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:06.315084 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" event={"ID":"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff","Type":"ContainerStarted","Data":"e54a923f12756bf8517f8f1df9f0abc5644034f75edf375cdbf87859f9bac493"} Jul 22 19:21:06.315473 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:06.315127 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" event={"ID":"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff","Type":"ContainerStarted","Data":"ff0c3aa49908f9c198fdbba7c154a072d3434e9f37fcdd90441cbf3d07968cd3"} Jul 22 19:21:06.333769 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:06.333722 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" podStartSLOduration=2.333706844 podStartE2EDuration="2.333706844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:21:06.331504834 +0000 UTC m=+2476.778900066" watchObservedRunningTime="2026-07-22 19:21:06.333706844 +0000 UTC m=+2476.781102076" Jul 22 19:21:15.298558 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:15.298526 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:15.299133 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:15.298575 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:15.311032 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:15.311005 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:21:15.366874 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:21:15.366845 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:22:04.924718 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:04.924676 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:05.923843 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:05.923805 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh"] Jul 22 19:22:05.945660 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:05.945633 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:06.676956 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:06.676920 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:06.890953 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:06.890923 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:16.579245 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:16.579203 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:20.944455 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:20.944412 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" podUID="1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff" containerName="main" containerID="cri-o://e54a923f12756bf8517f8f1df9f0abc5644034f75edf375cdbf87859f9bac493" gracePeriod=45 Jul 22 19:22:21.139878 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.139842 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:21.192964 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.192936 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:22:21.301136 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.301103 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dfd9\" (UniqueName: \"kubernetes.io/projected/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-kube-api-access-9dfd9\") pod \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " Jul 22 19:22:21.301136 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.301143 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-model-cache\") pod \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " Jul 22 19:22:21.301366 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.301296 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-home\") pod \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " Jul 22 19:22:21.301366 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.301336 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-dshm\") pod \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " Jul 22 19:22:21.301479 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.301366 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-tls-certs\") pod \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " Jul 22 19:22:21.301479 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.301427 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff" (UID: "1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:22:21.301479 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.301432 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-tmp-dir\") pod \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\" (UID: \"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff\") " Jul 22 19:22:21.301637 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.301539 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-home" (OuterVolumeSpecName: "home") pod "1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff" (UID: "1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:22:21.301637 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.301609 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff" (UID: "1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:22:21.301771 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.301747 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:22:21.301771 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.301769 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:22:21.301894 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.301778 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:22:21.303531 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.303499 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-dshm" (OuterVolumeSpecName: "dshm") pod "1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff" (UID: "1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:22:21.303633 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.303561 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff" (UID: "1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:22:21.303633 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.303586 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-kube-api-access-9dfd9" (OuterVolumeSpecName: "kube-api-access-9dfd9") pod "1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff" (UID: "1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff"). InnerVolumeSpecName "kube-api-access-9dfd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:22:21.402708 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.402670 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:22:21.402708 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.402701 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:22:21.402708 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.402711 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dfd9\" (UniqueName: \"kubernetes.io/projected/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff-kube-api-access-9dfd9\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:22:21.558965 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.558879 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:21.658661 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.658629 2574 generic.go:356] "Generic (PLEG): container finished" podID="1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff" containerID="e54a923f12756bf8517f8f1df9f0abc5644034f75edf375cdbf87859f9bac493" exitCode=0 Jul 22 19:22:21.658820 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.658673 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" Jul 22 19:22:21.658820 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.658677 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" event={"ID":"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff","Type":"ContainerDied","Data":"e54a923f12756bf8517f8f1df9f0abc5644034f75edf375cdbf87859f9bac493"} Jul 22 19:22:21.658820 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.658767 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh" event={"ID":"1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff","Type":"ContainerDied","Data":"ff0c3aa49908f9c198fdbba7c154a072d3434e9f37fcdd90441cbf3d07968cd3"} Jul 22 19:22:21.658820 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.658783 2574 scope.go:117] "RemoveContainer" containerID="e54a923f12756bf8517f8f1df9f0abc5644034f75edf375cdbf87859f9bac493" Jul 22 19:22:21.667921 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.667899 2574 scope.go:117] "RemoveContainer" containerID="e54a923f12756bf8517f8f1df9f0abc5644034f75edf375cdbf87859f9bac493" Jul 22 19:22:21.668197 ip-10-0-143-15 kubenswrapper[2574]: E0722 19:22:21.668176 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e54a923f12756bf8517f8f1df9f0abc5644034f75edf375cdbf87859f9bac493\": container with ID starting with e54a923f12756bf8517f8f1df9f0abc5644034f75edf375cdbf87859f9bac493 not found: ID does not exist" containerID="e54a923f12756bf8517f8f1df9f0abc5644034f75edf375cdbf87859f9bac493" Jul 22 19:22:21.668283 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.668229 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e54a923f12756bf8517f8f1df9f0abc5644034f75edf375cdbf87859f9bac493"} err="failed to get container status \"e54a923f12756bf8517f8f1df9f0abc5644034f75edf375cdbf87859f9bac493\": rpc error: code = NotFound desc = could not find container \"e54a923f12756bf8517f8f1df9f0abc5644034f75edf375cdbf87859f9bac493\": container with ID starting with e54a923f12756bf8517f8f1df9f0abc5644034f75edf375cdbf87859f9bac493 not found: ID does not exist" Jul 22 19:22:21.681355 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.681331 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh"] Jul 22 19:22:21.684106 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:21.684085 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ba9716/test-model-ba9716-kserve-77845bcc8f-pcnxh"] Jul 22 19:22:22.004868 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:22.004836 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff" path="/var/lib/kubelet/pods/1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff/volumes" Jul 22 19:22:27.228334 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:27.228298 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:27.904250 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:27.904215 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj"] Jul 22 19:22:27.904568 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:27.904556 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff" containerName="main" Jul 22 19:22:27.904613 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:27.904570 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff" containerName="main" Jul 22 19:22:27.904648 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:27.904630 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1983cd9f-91c7-4679-ac2a-a4e51ae8f8ff" containerName="main" Jul 22 19:22:27.909074 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:27.909057 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj" Jul 22 19:22:27.911234 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:27.911212 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 19:22:27.911342 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:27.911214 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r5ttk\"" Jul 22 19:22:27.914247 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:27.914227 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj"] Jul 22 19:22:28.058397 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:28.058361 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c677b1-63af-48ef-b424-058dae08e7f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj\" (UID: \"80c677b1-63af-48ef-b424-058dae08e7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj" Jul 22 19:22:28.159794 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:28.159715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c677b1-63af-48ef-b424-058dae08e7f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj\" (UID: \"80c677b1-63af-48ef-b424-058dae08e7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj" Jul 22 19:22:28.160351 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:28.160330 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c677b1-63af-48ef-b424-058dae08e7f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj\" (UID: \"80c677b1-63af-48ef-b424-058dae08e7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj" Jul 22 19:22:28.189694 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:28.189668 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:28.220526 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:28.220505 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj" Jul 22 19:22:28.292259 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:28.292230 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:28.352143 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:28.352121 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj"] Jul 22 19:22:28.354262 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:22:28.354237 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c677b1_63af_48ef_b424_058dae08e7f6.slice/crio-a2c3e73abcf7e6550874e54130f7c490455c982b7a1ee8f7ce36295e5d770f4c WatchSource:0}: Error finding container a2c3e73abcf7e6550874e54130f7c490455c982b7a1ee8f7ce36295e5d770f4c: Status 404 returned error can't find the container with id a2c3e73abcf7e6550874e54130f7c490455c982b7a1ee8f7ce36295e5d770f4c Jul 22 19:22:28.403156 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:28.403129 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:28.502219 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:28.502190 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:28.694725 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:28.694691 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj" event={"ID":"80c677b1-63af-48ef-b424-058dae08e7f6","Type":"ContainerStarted","Data":"8f30e3feaca8a815c8b8a09e4a2587bfc97890800f8207528fb1296226800c72"} Jul 22 19:22:28.694830 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:28.694784 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj" event={"ID":"80c677b1-63af-48ef-b424-058dae08e7f6","Type":"ContainerStarted","Data":"a2c3e73abcf7e6550874e54130f7c490455c982b7a1ee8f7ce36295e5d770f4c"} Jul 22 19:22:28.710446 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:28.710402 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj" podStartSLOduration=1.710387753 podStartE2EDuration="1.710387753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:22:28.708813951 +0000 UTC m=+2559.156209183" watchObservedRunningTime="2026-07-22 19:22:28.710387753 +0000 UTC m=+2559.157782984" Jul 22 19:22:29.700891 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:29.700854 2574 generic.go:356] "Generic (PLEG): container finished" podID="80c677b1-63af-48ef-b424-058dae08e7f6" containerID="8f30e3feaca8a815c8b8a09e4a2587bfc97890800f8207528fb1296226800c72" exitCode=0 Jul 22 19:22:29.701345 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:29.700936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj" event={"ID":"80c677b1-63af-48ef-b424-058dae08e7f6","Type":"ContainerDied","Data":"8f30e3feaca8a815c8b8a09e4a2587bfc97890800f8207528fb1296226800c72"} Jul 22 19:22:30.650081 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:30.650050 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:30.830110 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:30.830088 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj" Jul 22 19:22:30.982169 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:30.982146 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c677b1-63af-48ef-b424-058dae08e7f6-maas-api-service-ca\") pod \"80c677b1-63af-48ef-b424-058dae08e7f6\" (UID: \"80c677b1-63af-48ef-b424-058dae08e7f6\") " Jul 22 19:22:30.982473 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:30.982452 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c677b1-63af-48ef-b424-058dae08e7f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80c677b1-63af-48ef-b424-058dae08e7f6" (UID: "80c677b1-63af-48ef-b424-058dae08e7f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:22:31.082784 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:31.082758 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c677b1-63af-48ef-b424-058dae08e7f6-maas-api-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:22:31.710857 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:31.710826 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj" event={"ID":"80c677b1-63af-48ef-b424-058dae08e7f6","Type":"ContainerDied","Data":"a2c3e73abcf7e6550874e54130f7c490455c982b7a1ee8f7ce36295e5d770f4c"} Jul 22 19:22:31.711052 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:31.710874 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2c3e73abcf7e6550874e54130f7c490455c982b7a1ee8f7ce36295e5d770f4c" Jul 22 19:22:31.711052 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:31.710842 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj" Jul 22 19:22:38.443852 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:38.443815 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:42.981910 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:42.981872 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj"] Jul 22 19:22:42.985380 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:42.985355 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba9716-kqvgj"] Jul 22 19:22:42.986000 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:42.985964 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:43.417004 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:43.416916 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:44.005279 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:44.005238 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80c677b1-63af-48ef-b424-058dae08e7f6" path="/var/lib/kubelet/pods/80c677b1-63af-48ef-b424-058dae08e7f6/volumes" Jul 22 19:22:44.632808 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:44.632767 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:49.182123 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:49.182094 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:49.240769 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:49.240740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:49.464447 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:49.464365 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:49.706154 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:49.706119 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:50.076457 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:50.076417 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:50.411658 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:50.411559 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:50.650715 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:50.650661 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:53.343335 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:53.343303 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:53.433323 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:53.433287 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:53.503860 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:53.503829 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:53.604481 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:53.604407 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:53.944440 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:53.944349 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:54.045012 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:54.044951 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:57.818643 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:57.818607 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:58.294288 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:58.294255 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:58.629269 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:58.629187 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:22:59.085441 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:22:59.085403 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:01.321379 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:01.321344 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:01.412786 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:01.412751 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:01.484894 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:01.484861 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:01.583907 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:01.583827 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:02.031890 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:02.031855 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:02.132705 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:02.132676 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:05.877175 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:05.877139 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:14.118190 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.118101 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6"] Jul 22 19:23:14.118620 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.118604 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c677b1-63af-48ef-b424-058dae08e7f6" containerName="revoke-keys" Jul 22 19:23:14.118692 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.118624 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c677b1-63af-48ef-b424-058dae08e7f6" containerName="revoke-keys" Jul 22 19:23:14.118745 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.118736 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="80c677b1-63af-48ef-b424-058dae08e7f6" containerName="revoke-keys" Jul 22 19:23:14.121739 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.121715 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6" Jul 22 19:23:14.124235 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.124215 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r5ttk\"" Jul 22 19:23:14.124340 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.124274 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 19:23:14.128222 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.128201 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6"] Jul 22 19:23:14.235880 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.235855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e038489-cd3d-4eb6-9acc-923583124d5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6\" (UID: \"5e038489-cd3d-4eb6-9acc-923583124d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6" Jul 22 19:23:14.336902 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.336874 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e038489-cd3d-4eb6-9acc-923583124d5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6\" (UID: \"5e038489-cd3d-4eb6-9acc-923583124d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6" Jul 22 19:23:14.337446 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.337428 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e038489-cd3d-4eb6-9acc-923583124d5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6\" (UID: \"5e038489-cd3d-4eb6-9acc-923583124d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6" Jul 22 19:23:14.432607 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.432555 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6" Jul 22 19:23:14.557848 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.557826 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6"] Jul 22 19:23:14.559818 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:23:14.559792 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e038489_cd3d_4eb6_9acc_923583124d5b.slice/crio-ba3f78755046e994e44d5d25905ce15f71cf7be301c794ee16c148a9fb77e73a WatchSource:0}: Error finding container ba3f78755046e994e44d5d25905ce15f71cf7be301c794ee16c148a9fb77e73a: Status 404 returned error can't find the container with id ba3f78755046e994e44d5d25905ce15f71cf7be301c794ee16c148a9fb77e73a Jul 22 19:23:14.597372 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.597342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:14.699777 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.699684 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:14.903442 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.903411 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6" event={"ID":"5e038489-cd3d-4eb6-9acc-923583124d5b","Type":"ContainerStarted","Data":"cdeb05114b76e4b6dd1cd0916e751f7e43cae427b1d16b9d7619c7104730faf4"} Jul 22 19:23:14.903579 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.903468 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6" event={"ID":"5e038489-cd3d-4eb6-9acc-923583124d5b","Type":"ContainerStarted","Data":"ba3f78755046e994e44d5d25905ce15f71cf7be301c794ee16c148a9fb77e73a"} Jul 22 19:23:14.919527 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:14.919485 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6" podStartSLOduration=0.919470384 podStartE2EDuration="919.470384ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:23:14.917600467 +0000 UTC m=+2605.364995690" watchObservedRunningTime="2026-07-22 19:23:14.919470384 +0000 UTC m=+2605.366865615" Jul 22 19:23:15.908985 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:15.908939 2574 generic.go:356] "Generic (PLEG): container finished" podID="5e038489-cd3d-4eb6-9acc-923583124d5b" containerID="cdeb05114b76e4b6dd1cd0916e751f7e43cae427b1d16b9d7619c7104730faf4" exitCode=0 Jul 22 19:23:15.909323 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:15.909029 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6" event={"ID":"5e038489-cd3d-4eb6-9acc-923583124d5b","Type":"ContainerDied","Data":"cdeb05114b76e4b6dd1cd0916e751f7e43cae427b1d16b9d7619c7104730faf4"} Jul 22 19:23:17.040999 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:17.040958 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6" Jul 22 19:23:17.158175 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:17.158146 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e038489-cd3d-4eb6-9acc-923583124d5b-maas-api-service-ca\") pod \"5e038489-cd3d-4eb6-9acc-923583124d5b\" (UID: \"5e038489-cd3d-4eb6-9acc-923583124d5b\") " Jul 22 19:23:17.158521 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:17.158494 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e038489-cd3d-4eb6-9acc-923583124d5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e038489-cd3d-4eb6-9acc-923583124d5b" (UID: "5e038489-cd3d-4eb6-9acc-923583124d5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:23:17.259546 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:17.259518 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e038489-cd3d-4eb6-9acc-923583124d5b-maas-api-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:23:17.919556 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:17.919525 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6" Jul 22 19:23:17.919732 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:17.919567 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6" event={"ID":"5e038489-cd3d-4eb6-9acc-923583124d5b","Type":"ContainerDied","Data":"ba3f78755046e994e44d5d25905ce15f71cf7be301c794ee16c148a9fb77e73a"} Jul 22 19:23:17.919732 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:17.919624 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba3f78755046e994e44d5d25905ce15f71cf7be301c794ee16c148a9fb77e73a" Jul 22 19:23:23.032100 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:23.032067 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:23.134285 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:23.134253 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:24.677738 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:24.677704 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:24.781339 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:24.781310 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:29.935261 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:29.935226 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:34.195138 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:34.195104 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6"] Jul 22 19:23:34.200097 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:34.200070 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:34.200567 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:34.200549 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6c9526-tkwq6"] Jul 22 19:23:36.004629 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:36.004596 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e038489-cd3d-4eb6-9acc-923583124d5b" path="/var/lib/kubelet/pods/5e038489-cd3d-4eb6-9acc-923583124d5b/volumes" Jul 22 19:23:37.334500 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:37.334465 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:40.314768 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:40.314734 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:40.414813 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:40.414776 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:40.597839 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:40.597762 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:40.840041 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:40.840009 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:40.971871 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:40.971801 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx"] Jul 22 19:23:40.972184 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:40.972171 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e038489-cd3d-4eb6-9acc-923583124d5b" containerName="revoke-keys" Jul 22 19:23:40.972234 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:40.972186 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e038489-cd3d-4eb6-9acc-923583124d5b" containerName="revoke-keys" Jul 22 19:23:40.972276 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:40.972244 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e038489-cd3d-4eb6-9acc-923583124d5b" containerName="revoke-keys" Jul 22 19:23:40.975447 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:40.975430 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx" Jul 22 19:23:40.977879 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:40.977854 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r5ttk\"" Jul 22 19:23:40.978016 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:40.977859 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 19:23:40.982276 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:40.982199 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx"] Jul 22 19:23:41.058998 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:41.058953 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/451f7f4d-9f5a-4e3c-a46e-66c28645389b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65c717-bf8nx\" (UID: \"451f7f4d-9f5a-4e3c-a46e-66c28645389b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx" Jul 22 19:23:41.159983 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:41.159953 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/451f7f4d-9f5a-4e3c-a46e-66c28645389b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65c717-bf8nx\" (UID: \"451f7f4d-9f5a-4e3c-a46e-66c28645389b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx" Jul 22 19:23:41.160669 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:41.160647 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/451f7f4d-9f5a-4e3c-a46e-66c28645389b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65c717-bf8nx\" (UID: \"451f7f4d-9f5a-4e3c-a46e-66c28645389b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx" Jul 22 19:23:41.286869 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:41.286844 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx" Jul 22 19:23:41.415352 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:41.415327 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx"] Jul 22 19:23:41.417821 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:23:41.417788 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451f7f4d_9f5a_4e3c_a46e_66c28645389b.slice/crio-31f48e246dacc4687e1fc30555997dc2fb4e89477538be3460b7abd8e2cc67fd WatchSource:0}: Error finding container 31f48e246dacc4687e1fc30555997dc2fb4e89477538be3460b7abd8e2cc67fd: Status 404 returned error can't find the container with id 31f48e246dacc4687e1fc30555997dc2fb4e89477538be3460b7abd8e2cc67fd Jul 22 19:23:41.659494 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:41.659464 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:41.760014 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:41.759964 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:42.025420 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:42.025387 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx" event={"ID":"451f7f4d-9f5a-4e3c-a46e-66c28645389b","Type":"ContainerStarted","Data":"af927ebff4d81e507c21f70de702051abb9719ebb7e8b2b24f9720ed8a31f367"} Jul 22 19:23:42.025589 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:42.025432 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx" event={"ID":"451f7f4d-9f5a-4e3c-a46e-66c28645389b","Type":"ContainerStarted","Data":"31f48e246dacc4687e1fc30555997dc2fb4e89477538be3460b7abd8e2cc67fd"} Jul 22 19:23:42.042856 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:42.040666 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx" podStartSLOduration=2.040645354 podStartE2EDuration="2.040645354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:23:42.039604824 +0000 UTC m=+2632.487000057" watchObservedRunningTime="2026-07-22 19:23:42.040645354 +0000 UTC m=+2632.488040587" Jul 22 19:23:43.031703 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:43.031668 2574 generic.go:356] "Generic (PLEG): container finished" podID="451f7f4d-9f5a-4e3c-a46e-66c28645389b" containerID="af927ebff4d81e507c21f70de702051abb9719ebb7e8b2b24f9720ed8a31f367" exitCode=0 Jul 22 19:23:43.032096 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:43.031724 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx" event={"ID":"451f7f4d-9f5a-4e3c-a46e-66c28645389b","Type":"ContainerDied","Data":"af927ebff4d81e507c21f70de702051abb9719ebb7e8b2b24f9720ed8a31f367"} Jul 22 19:23:44.166516 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:44.166491 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx" Jul 22 19:23:44.285084 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:44.285054 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/451f7f4d-9f5a-4e3c-a46e-66c28645389b-maas-api-service-ca\") pod \"451f7f4d-9f5a-4e3c-a46e-66c28645389b\" (UID: \"451f7f4d-9f5a-4e3c-a46e-66c28645389b\") " Jul 22 19:23:44.285410 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:44.285385 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451f7f4d-9f5a-4e3c-a46e-66c28645389b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "451f7f4d-9f5a-4e3c-a46e-66c28645389b" (UID: "451f7f4d-9f5a-4e3c-a46e-66c28645389b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:23:44.385536 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:44.385475 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/451f7f4d-9f5a-4e3c-a46e-66c28645389b-maas-api-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:23:45.042030 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:45.041964 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx" event={"ID":"451f7f4d-9f5a-4e3c-a46e-66c28645389b","Type":"ContainerDied","Data":"31f48e246dacc4687e1fc30555997dc2fb4e89477538be3460b7abd8e2cc67fd"} Jul 22 19:23:45.042203 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:45.042040 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31f48e246dacc4687e1fc30555997dc2fb4e89477538be3460b7abd8e2cc67fd" Jul 22 19:23:45.042203 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:45.041997 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx" Jul 22 19:23:45.543723 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:45.543683 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:45.645467 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:45.645431 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:51.601278 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:51.601238 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:51.701593 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:51.701547 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:23:56.863696 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:23:56.863651 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:01.045308 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:01.045274 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:01.046591 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:01.046565 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx"] Jul 22 19:24:01.048113 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:01.048092 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65c717-bf8nx"] Jul 22 19:24:01.530622 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:01.530586 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:02.004876 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:02.004847 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="451f7f4d-9f5a-4e3c-a46e-66c28645389b" path="/var/lib/kubelet/pods/451f7f4d-9f5a-4e3c-a46e-66c28645389b/volumes" Jul 22 19:24:07.364134 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:07.364097 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:07.462418 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:07.462392 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:07.649165 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:07.649068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:07.897820 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:07.897786 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:08.410966 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:08.410930 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:08.590377 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:08.590331 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:08.800385 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:08.800355 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:11.721526 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:11.721484 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:11.785199 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:11.785162 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:11.915023 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:11.914959 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:12.414111 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:12.414080 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:12.579667 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:12.579632 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:12.679122 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:12.679038 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:16.284780 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:16.284741 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:16.659303 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:16.659201 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:16.974604 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:16.974572 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:17.373444 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:17.373362 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:19.819216 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:19.819185 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:19.907045 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:19.907007 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:19.979150 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:19.979116 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:20.080164 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:20.080068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:20.468531 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:20.468447 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:20.574881 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:20.574847 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:24.419053 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:24.419008 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:34.454492 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:34.454459 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:34.554267 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:34.554228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:41.309929 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:41.309886 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:41.414245 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:41.414186 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:42.405065 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:42.405031 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:42.454140 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:42.454107 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:45.398063 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.397961 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j"] Jul 22 19:24:45.398469 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.398331 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="451f7f4d-9f5a-4e3c-a46e-66c28645389b" containerName="revoke-keys" Jul 22 19:24:45.398469 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.398343 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="451f7f4d-9f5a-4e3c-a46e-66c28645389b" containerName="revoke-keys" Jul 22 19:24:45.398469 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.398410 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="451f7f4d-9f5a-4e3c-a46e-66c28645389b" containerName="revoke-keys" Jul 22 19:24:45.401617 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.401597 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.403959 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.403926 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4ac498\"/\"auth-test-model-4ac498-kserve-self-signed-certs\"" Jul 22 19:24:45.404637 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.404613 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4ac498\"/\"default-dockercfg-gskw5\"" Jul 22 19:24:45.404754 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.404680 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4ac498\"/\"kube-root-ca.crt\"" Jul 22 19:24:45.404754 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.404682 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4ac498\"/\"openshift-service-ca.crt\"" Jul 22 19:24:45.409642 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.409621 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j"] Jul 22 19:24:45.488625 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.488584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-tls-certs\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.488625 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.488624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-home\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.488863 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.488641 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-model-cache\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.488863 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.488760 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-dshm\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.488863 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.488795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw5p6\" (UniqueName: \"kubernetes.io/projected/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-kube-api-access-vw5p6\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.488863 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.488840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-tmp-dir\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.501742 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.501710 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:24:45.589666 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.589627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-tls-certs\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.589666 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.589668 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-home\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.589902 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.589686 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-model-cache\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.589902 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.589748 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-dshm\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.589902 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.589769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw5p6\" (UniqueName: \"kubernetes.io/projected/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-kube-api-access-vw5p6\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.589902 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.589806 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-tmp-dir\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.590159 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.590105 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-home\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.590218 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.590170 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-model-cache\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.590218 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.590192 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-tmp-dir\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.592007 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.591986 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-dshm\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.592304 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.592282 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-tls-certs\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.598350 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.598327 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw5p6\" (UniqueName: \"kubernetes.io/projected/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-kube-api-access-vw5p6\") pod \"auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.711808 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.711720 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:45.841153 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:45.841127 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j"] Jul 22 19:24:45.843049 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:24:45.843024 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5e1783f_1f1d_4a4e_a055_67ed746f9b5a.slice/crio-d501f9e410e3913552bc3001dd36e7d16202b71110c10d8455c8a1824c6a25b2 WatchSource:0}: Error finding container d501f9e410e3913552bc3001dd36e7d16202b71110c10d8455c8a1824c6a25b2: Status 404 returned error can't find the container with id d501f9e410e3913552bc3001dd36e7d16202b71110c10d8455c8a1824c6a25b2 Jul 22 19:24:46.337095 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:46.337066 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" event={"ID":"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a","Type":"ContainerStarted","Data":"c45b96f86a89b998d04d150c937e574ae33c1f537d94ffc0bfb0ac821ce91c70"} Jul 22 19:24:46.337316 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:46.337102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" event={"ID":"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a","Type":"ContainerStarted","Data":"d501f9e410e3913552bc3001dd36e7d16202b71110c10d8455c8a1824c6a25b2"} Jul 22 19:24:46.360910 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:46.360861 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" podStartSLOduration=1.360845959 podStartE2EDuration="1.360845959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:24:46.359762371 +0000 UTC m=+2696.807157602" watchObservedRunningTime="2026-07-22 19:24:46.360845959 +0000 UTC m=+2696.808241185" Jul 22 19:24:50.126940 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:50.126902 2574 scope.go:117] "RemoveContainer" containerID="12dbfbb097fb19f1d68e22decdaa33f61a466cc2bf405f667670b9ba5790c0ec" Jul 22 19:24:55.712621 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:55.712588 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:55.713038 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:55.712652 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:55.725291 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:55.725268 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:24:56.397729 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:24:56.397700 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:25:15.030517 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:25:15.030483 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:26:26.081821 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:26:26.081743 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:26:39.806934 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:26:39.806897 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:27:16.406337 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:16.406299 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j"] Jul 22 19:27:16.422005 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:16.421954 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:27:17.241878 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:17.241845 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:27:21.318595 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:21.318566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:27:31.427078 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.427041 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" podUID="a5e1783f-1f1d-4a4e-a055-67ed746f9b5a" containerName="main" containerID="cri-o://c45b96f86a89b998d04d150c937e574ae33c1f537d94ffc0bfb0ac821ce91c70" gracePeriod=45 Jul 22 19:27:31.677683 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.677627 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:27:31.777030 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.777003 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-tls-certs\") pod \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " Jul 22 19:27:31.777187 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.777062 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-dshm\") pod \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " Jul 22 19:27:31.777187 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.777083 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw5p6\" (UniqueName: \"kubernetes.io/projected/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-kube-api-access-vw5p6\") pod \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " Jul 22 19:27:31.777295 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.777188 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-model-cache\") pod \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " Jul 22 19:27:31.777354 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.777306 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-tmp-dir\") pod \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " Jul 22 19:27:31.777409 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.777368 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-home\") pod \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\" (UID: \"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a\") " Jul 22 19:27:31.777503 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.777472 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5e1783f-1f1d-4a4e-a055-67ed746f9b5a" (UID: "a5e1783f-1f1d-4a4e-a055-67ed746f9b5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:27:31.777643 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.777626 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:27:31.777690 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.777636 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5e1783f-1f1d-4a4e-a055-67ed746f9b5a" (UID: "a5e1783f-1f1d-4a4e-a055-67ed746f9b5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:27:31.777690 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.777647 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-home" (OuterVolumeSpecName: "home") pod "a5e1783f-1f1d-4a4e-a055-67ed746f9b5a" (UID: "a5e1783f-1f1d-4a4e-a055-67ed746f9b5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:27:31.779321 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.779290 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-kube-api-access-vw5p6" (OuterVolumeSpecName: "kube-api-access-vw5p6") pod "a5e1783f-1f1d-4a4e-a055-67ed746f9b5a" (UID: "a5e1783f-1f1d-4a4e-a055-67ed746f9b5a"). InnerVolumeSpecName "kube-api-access-vw5p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:27:31.779427 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.779329 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5e1783f-1f1d-4a4e-a055-67ed746f9b5a" (UID: "a5e1783f-1f1d-4a4e-a055-67ed746f9b5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:27:31.779427 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.779337 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-dshm" (OuterVolumeSpecName: "dshm") pod "a5e1783f-1f1d-4a4e-a055-67ed746f9b5a" (UID: "a5e1783f-1f1d-4a4e-a055-67ed746f9b5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:27:31.879023 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.878996 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:27:31.879023 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.879022 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:27:31.879194 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.879032 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:27:31.879194 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.879041 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:27:31.879194 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:31.879049 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw5p6\" (UniqueName: \"kubernetes.io/projected/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a-kube-api-access-vw5p6\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:27:32.113694 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:32.113667 2574 generic.go:356] "Generic (PLEG): container finished" podID="a5e1783f-1f1d-4a4e-a055-67ed746f9b5a" containerID="c45b96f86a89b998d04d150c937e574ae33c1f537d94ffc0bfb0ac821ce91c70" exitCode=0 Jul 22 19:27:32.113828 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:32.113708 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" Jul 22 19:27:32.113828 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:32.113757 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" event={"ID":"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a","Type":"ContainerDied","Data":"c45b96f86a89b998d04d150c937e574ae33c1f537d94ffc0bfb0ac821ce91c70"} Jul 22 19:27:32.113828 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:32.113797 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j" event={"ID":"a5e1783f-1f1d-4a4e-a055-67ed746f9b5a","Type":"ContainerDied","Data":"d501f9e410e3913552bc3001dd36e7d16202b71110c10d8455c8a1824c6a25b2"} Jul 22 19:27:32.113828 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:32.113818 2574 scope.go:117] "RemoveContainer" containerID="c45b96f86a89b998d04d150c937e574ae33c1f537d94ffc0bfb0ac821ce91c70" Jul 22 19:27:32.121924 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:32.121906 2574 scope.go:117] "RemoveContainer" containerID="c45b96f86a89b998d04d150c937e574ae33c1f537d94ffc0bfb0ac821ce91c70" Jul 22 19:27:32.122195 ip-10-0-143-15 kubenswrapper[2574]: E0722 19:27:32.122177 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c45b96f86a89b998d04d150c937e574ae33c1f537d94ffc0bfb0ac821ce91c70\": container with ID starting with c45b96f86a89b998d04d150c937e574ae33c1f537d94ffc0bfb0ac821ce91c70 not found: ID does not exist" containerID="c45b96f86a89b998d04d150c937e574ae33c1f537d94ffc0bfb0ac821ce91c70" Jul 22 19:27:32.122251 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:32.122206 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45b96f86a89b998d04d150c937e574ae33c1f537d94ffc0bfb0ac821ce91c70"} err="failed to get container status \"c45b96f86a89b998d04d150c937e574ae33c1f537d94ffc0bfb0ac821ce91c70\": rpc error: code = NotFound desc = could not find container \"c45b96f86a89b998d04d150c937e574ae33c1f537d94ffc0bfb0ac821ce91c70\": container with ID starting with c45b96f86a89b998d04d150c937e574ae33c1f537d94ffc0bfb0ac821ce91c70 not found: ID does not exist" Jul 22 19:27:32.148634 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:32.148608 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j"] Jul 22 19:27:32.162501 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:32.162474 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4ac498/auth-test-model-4ac498-kserve-6557bbb9c5-g5d5j"] Jul 22 19:27:34.004636 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:34.004601 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5e1783f-1f1d-4a4e-a055-67ed746f9b5a" path="/var/lib/kubelet/pods/a5e1783f-1f1d-4a4e-a055-67ed746f9b5a/volumes" Jul 22 19:27:53.893994 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.893933 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4"] Jul 22 19:27:53.894487 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.894443 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5e1783f-1f1d-4a4e-a055-67ed746f9b5a" containerName="main" Jul 22 19:27:53.894487 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.894464 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5e1783f-1f1d-4a4e-a055-67ed746f9b5a" containerName="main" Jul 22 19:27:53.894590 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.894577 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5e1783f-1f1d-4a4e-a055-67ed746f9b5a" containerName="main" Jul 22 19:27:53.899101 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.899077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:53.902323 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.902300 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8f2692\"/\"kube-root-ca.crt\"" Jul 22 19:27:53.902505 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.902313 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8f2692\"/\"sub-test-model-8f2692-kserve-self-signed-certs\"" Jul 22 19:27:53.902630 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.902368 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8f2692\"/\"openshift-service-ca.crt\"" Jul 22 19:27:53.902630 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.902401 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8f2692\"/\"default-dockercfg-h442p\"" Jul 22 19:27:53.905519 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.905496 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4"] Jul 22 19:27:53.922117 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.922095 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:27:53.967065 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.967043 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-model-cache\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:53.967193 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.967112 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-dshm\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:53.967193 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.967158 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29t8j\" (UniqueName: \"kubernetes.io/projected/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-kube-api-access-29t8j\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:53.967290 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.967207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-tmp-dir\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:53.967290 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.967241 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-home\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:53.967357 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:53.967291 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-tls-certs\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.068576 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.068548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-dshm\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.068726 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.068584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29t8j\" (UniqueName: \"kubernetes.io/projected/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-kube-api-access-29t8j\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.068726 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.068615 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-tmp-dir\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.068726 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.068651 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-home\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.068726 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.068679 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-tls-certs\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.068726 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.068706 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-model-cache\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.069113 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.069089 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-home\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.069190 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.069099 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-model-cache\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.069190 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.069153 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-tmp-dir\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.070927 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.070906 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-dshm\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.071268 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.071250 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-tls-certs\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.076739 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.076720 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29t8j\" (UniqueName: \"kubernetes.io/projected/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-kube-api-access-29t8j\") pod \"sub-test-model-8f2692-kserve-6f875d8bbf-86vk4\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.209162 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.209095 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:27:54.341258 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:54.341233 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4"] Jul 22 19:27:54.343243 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:27:54.343215 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod798f2d5d_99dc_4288_8ed2_9b9dfb9127b2.slice/crio-72db2bc0709ceda3a5ad6577821f02ae0b5ab942b9ea0e81cce9243469feb9c0 WatchSource:0}: Error finding container 72db2bc0709ceda3a5ad6577821f02ae0b5ab942b9ea0e81cce9243469feb9c0: Status 404 returned error can't find the container with id 72db2bc0709ceda3a5ad6577821f02ae0b5ab942b9ea0e81cce9243469feb9c0 Jul 22 19:27:55.224338 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:55.224305 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" event={"ID":"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2","Type":"ContainerStarted","Data":"c5b840cf14ec12780dc3cf342134f982f9acfea595f62edfc05997b2b278f3cf"} Jul 22 19:27:55.224805 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:55.224349 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" event={"ID":"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2","Type":"ContainerStarted","Data":"72db2bc0709ceda3a5ad6577821f02ae0b5ab942b9ea0e81cce9243469feb9c0"} Jul 22 19:27:55.243723 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:55.243675 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" podStartSLOduration=2.243659924 podStartE2EDuration="2.243659924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:27:55.241244893 +0000 UTC m=+2885.688640125" watchObservedRunningTime="2026-07-22 19:27:55.243659924 +0000 UTC m=+2885.691055158" Jul 22 19:27:59.978247 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:27:59.978208 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:28:00.086191 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:28:00.086161 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:28:04.209827 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:28:04.209792 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:28:04.210317 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:28:04.209842 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:28:04.222272 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:28:04.222251 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:28:04.276362 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:28:04.276335 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:28:46.273471 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:28:46.273430 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:28:47.897991 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:28:47.897937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:28:50.175391 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:28:50.175362 2574 scope.go:117] "RemoveContainer" containerID="8f30e3feaca8a815c8b8a09e4a2587bfc97890800f8207528fb1296226800c72" Jul 22 19:28:56.612413 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:28:56.612368 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:28:57.475313 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:28:57.475277 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4"] Jul 22 19:28:57.494198 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:28:57.494155 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:29:01.659455 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:01.659417 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:29:12.497445 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.497406 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" podUID="798f2d5d-99dc-4288-8ed2-9b9dfb9127b2" containerName="main" containerID="cri-o://c5b840cf14ec12780dc3cf342134f982f9acfea595f62edfc05997b2b278f3cf" gracePeriod=45 Jul 22 19:29:12.749845 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.749786 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:29:12.905222 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.905139 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-dshm\") pod \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " Jul 22 19:29:12.905222 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.905185 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-tls-certs\") pod \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " Jul 22 19:29:12.905442 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.905260 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29t8j\" (UniqueName: \"kubernetes.io/projected/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-kube-api-access-29t8j\") pod \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " Jul 22 19:29:12.905442 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.905297 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-model-cache\") pod \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " Jul 22 19:29:12.905442 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.905331 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-home\") pod \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " Jul 22 19:29:12.905442 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.905366 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-tmp-dir\") pod \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\" (UID: \"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2\") " Jul 22 19:29:12.905634 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.905599 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-home" (OuterVolumeSpecName: "home") pod "798f2d5d-99dc-4288-8ed2-9b9dfb9127b2" (UID: "798f2d5d-99dc-4288-8ed2-9b9dfb9127b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:29:12.905704 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.905613 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "798f2d5d-99dc-4288-8ed2-9b9dfb9127b2" (UID: "798f2d5d-99dc-4288-8ed2-9b9dfb9127b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:29:12.905704 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.905675 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "798f2d5d-99dc-4288-8ed2-9b9dfb9127b2" (UID: "798f2d5d-99dc-4288-8ed2-9b9dfb9127b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:29:12.907353 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.907326 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-dshm" (OuterVolumeSpecName: "dshm") pod "798f2d5d-99dc-4288-8ed2-9b9dfb9127b2" (UID: "798f2d5d-99dc-4288-8ed2-9b9dfb9127b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:29:12.907476 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.907415 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-kube-api-access-29t8j" (OuterVolumeSpecName: "kube-api-access-29t8j") pod "798f2d5d-99dc-4288-8ed2-9b9dfb9127b2" (UID: "798f2d5d-99dc-4288-8ed2-9b9dfb9127b2"). InnerVolumeSpecName "kube-api-access-29t8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:29:12.907476 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:12.907418 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "798f2d5d-99dc-4288-8ed2-9b9dfb9127b2" (UID: "798f2d5d-99dc-4288-8ed2-9b9dfb9127b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:29:13.006592 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.006526 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29t8j\" (UniqueName: \"kubernetes.io/projected/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-kube-api-access-29t8j\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:29:13.006592 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.006556 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:29:13.006592 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.006566 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:29:13.006592 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.006594 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:29:13.006592 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.006602 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:29:13.006827 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.006610 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:29:13.589568 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.589536 2574 generic.go:356] "Generic (PLEG): container finished" podID="798f2d5d-99dc-4288-8ed2-9b9dfb9127b2" containerID="c5b840cf14ec12780dc3cf342134f982f9acfea595f62edfc05997b2b278f3cf" exitCode=0 Jul 22 19:29:13.589950 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.589580 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" Jul 22 19:29:13.589950 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.589621 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" event={"ID":"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2","Type":"ContainerDied","Data":"c5b840cf14ec12780dc3cf342134f982f9acfea595f62edfc05997b2b278f3cf"} Jul 22 19:29:13.589950 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.589664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4" event={"ID":"798f2d5d-99dc-4288-8ed2-9b9dfb9127b2","Type":"ContainerDied","Data":"72db2bc0709ceda3a5ad6577821f02ae0b5ab942b9ea0e81cce9243469feb9c0"} Jul 22 19:29:13.589950 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.589685 2574 scope.go:117] "RemoveContainer" containerID="c5b840cf14ec12780dc3cf342134f982f9acfea595f62edfc05997b2b278f3cf" Jul 22 19:29:13.599128 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.599113 2574 scope.go:117] "RemoveContainer" containerID="c5b840cf14ec12780dc3cf342134f982f9acfea595f62edfc05997b2b278f3cf" Jul 22 19:29:13.599365 ip-10-0-143-15 kubenswrapper[2574]: E0722 19:29:13.599347 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b840cf14ec12780dc3cf342134f982f9acfea595f62edfc05997b2b278f3cf\": container with ID starting with c5b840cf14ec12780dc3cf342134f982f9acfea595f62edfc05997b2b278f3cf not found: ID does not exist" containerID="c5b840cf14ec12780dc3cf342134f982f9acfea595f62edfc05997b2b278f3cf" Jul 22 19:29:13.599412 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.599372 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b840cf14ec12780dc3cf342134f982f9acfea595f62edfc05997b2b278f3cf"} err="failed to get container status \"c5b840cf14ec12780dc3cf342134f982f9acfea595f62edfc05997b2b278f3cf\": rpc error: code = NotFound desc = could not find container \"c5b840cf14ec12780dc3cf342134f982f9acfea595f62edfc05997b2b278f3cf\": container with ID starting with c5b840cf14ec12780dc3cf342134f982f9acfea595f62edfc05997b2b278f3cf not found: ID does not exist" Jul 22 19:29:13.610936 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.610913 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4"] Jul 22 19:29:13.614794 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:13.614776 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-8f2692/sub-test-model-8f2692-kserve-6f875d8bbf-86vk4"] Jul 22 19:29:14.005447 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:14.005415 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="798f2d5d-99dc-4288-8ed2-9b9dfb9127b2" path="/var/lib/kubelet/pods/798f2d5d-99dc-4288-8ed2-9b9dfb9127b2/volumes" Jul 22 19:29:17.681495 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:17.681457 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:29:35.218431 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:35.218392 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:29:35.317839 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:35.317800 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:29:42.027599 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:42.027558 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:29:42.128635 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:42.128600 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:29:44.888601 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.888567 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb"] Jul 22 19:29:44.889075 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.888937 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="798f2d5d-99dc-4288-8ed2-9b9dfb9127b2" containerName="main" Jul 22 19:29:44.889075 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.888948 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="798f2d5d-99dc-4288-8ed2-9b9dfb9127b2" containerName="main" Jul 22 19:29:44.889075 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.889037 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="798f2d5d-99dc-4288-8ed2-9b9dfb9127b2" containerName="main" Jul 22 19:29:44.892054 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.892034 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:44.894216 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.894195 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8f2692\"/\"default-dockercfg-h442p\"" Jul 22 19:29:44.895072 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.895051 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8f2692\"/\"rate-test-model-8f2692-kserve-self-signed-certs\"" Jul 22 19:29:44.895180 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.895081 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8f2692\"/\"kube-root-ca.crt\"" Jul 22 19:29:44.895180 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.895125 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8f2692\"/\"openshift-service-ca.crt\"" Jul 22 19:29:44.900561 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.900540 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb"] Jul 22 19:29:44.913330 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.913303 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:29:44.942431 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.942407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a9876f-a833-490c-8a28-3f62dcb1ea2b-tls-certs\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:44.942519 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.942449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-dshm\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:44.942560 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.942523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-tmp-dir\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:44.942595 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.942560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-model-cache\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:44.942595 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.942579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-home\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:44.942663 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:44.942606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxcdn\" (UniqueName: \"kubernetes.io/projected/08a9876f-a833-490c-8a28-3f62dcb1ea2b-kube-api-access-wxcdn\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.043745 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.043717 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-dshm\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.043854 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.043823 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-tmp-dir\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.044004 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.043876 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-model-cache\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.044004 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.043919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-home\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.044004 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.043955 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxcdn\" (UniqueName: \"kubernetes.io/projected/08a9876f-a833-490c-8a28-3f62dcb1ea2b-kube-api-access-wxcdn\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.044175 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.044013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a9876f-a833-490c-8a28-3f62dcb1ea2b-tls-certs\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.044340 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.044314 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-model-cache\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.044437 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.044353 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-home\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.044544 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.044524 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-tmp-dir\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.046134 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.046115 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-dshm\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.046429 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.046411 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a9876f-a833-490c-8a28-3f62dcb1ea2b-tls-certs\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.051156 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.051133 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxcdn\" (UniqueName: \"kubernetes.io/projected/08a9876f-a833-490c-8a28-3f62dcb1ea2b-kube-api-access-wxcdn\") pod \"rate-test-model-8f2692-kserve-94c5865bc-4j2nb\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.202243 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.202173 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:45.329471 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.329380 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb"] Jul 22 19:29:45.332260 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:29:45.332235 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08a9876f_a833_490c_8a28_3f62dcb1ea2b.slice/crio-04e3e4648422791702d3e2fb53fb0ca9ac091e33f1d8c0e6eb7ac9f5faad628a WatchSource:0}: Error finding container 04e3e4648422791702d3e2fb53fb0ca9ac091e33f1d8c0e6eb7ac9f5faad628a: Status 404 returned error can't find the container with id 04e3e4648422791702d3e2fb53fb0ca9ac091e33f1d8c0e6eb7ac9f5faad628a Jul 22 19:29:45.729547 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.729510 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" event={"ID":"08a9876f-a833-490c-8a28-3f62dcb1ea2b","Type":"ContainerStarted","Data":"1400b808245d61cc389143045bc45175ea515e20f584e19454ff770f3a8b2a13"} Jul 22 19:29:45.729720 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.729558 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" event={"ID":"08a9876f-a833-490c-8a28-3f62dcb1ea2b","Type":"ContainerStarted","Data":"04e3e4648422791702d3e2fb53fb0ca9ac091e33f1d8c0e6eb7ac9f5faad628a"} Jul 22 19:29:45.744197 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:45.744150 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" podStartSLOduration=1.7441358089999999 podStartE2EDuration="1.744135809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:29:45.743020503 +0000 UTC m=+2996.190415728" watchObservedRunningTime="2026-07-22 19:29:45.744135809 +0000 UTC m=+2996.191531041" Jul 22 19:29:50.195895 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:50.195854 2574 scope.go:117] "RemoveContainer" containerID="af927ebff4d81e507c21f70de702051abb9719ebb7e8b2b24f9720ed8a31f367" Jul 22 19:29:50.203848 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:50.203825 2574 scope.go:117] "RemoveContainer" containerID="cdeb05114b76e4b6dd1cd0916e751f7e43cae427b1d16b9d7619c7104730faf4" Jul 22 19:29:55.202837 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:55.202797 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:55.203293 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:55.202867 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:55.215131 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:55.215110 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:29:55.784527 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:29:55.784493 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:30:21.667246 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:21.667213 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:30:21.770593 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:21.770564 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:30:26.723984 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:26.723937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:30:27.379177 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:27.379137 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb"] Jul 22 19:30:27.397225 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:27.397193 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:30:35.272491 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:35.272453 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7789-140041791329312"] Jul 22 19:30:35.275982 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:35.275955 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:35.278120 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:35.278096 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m4ctx\"" Jul 22 19:30:35.281306 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:35.281282 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140041791329312"] Jul 22 19:30:35.361382 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:35.361351 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rpp6\" (UniqueName: \"kubernetes.io/projected/148b41c5-11e6-4fcf-9bd8-34f08d15f184-kube-api-access-9rpp6\") pod \"test-curl-7789-140041791329312\" (UID: \"148b41c5-11e6-4fcf-9bd8-34f08d15f184\") " pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:35.462014 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:35.461987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rpp6\" (UniqueName: \"kubernetes.io/projected/148b41c5-11e6-4fcf-9bd8-34f08d15f184-kube-api-access-9rpp6\") pod \"test-curl-7789-140041791329312\" (UID: \"148b41c5-11e6-4fcf-9bd8-34f08d15f184\") " pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:35.470058 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:35.470029 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rpp6\" (UniqueName: \"kubernetes.io/projected/148b41c5-11e6-4fcf-9bd8-34f08d15f184-kube-api-access-9rpp6\") pod \"test-curl-7789-140041791329312\" (UID: \"148b41c5-11e6-4fcf-9bd8-34f08d15f184\") " pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:35.586620 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:35.586532 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:35.716624 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:35.716590 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140041791329312"] Jul 22 19:30:35.719134 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:30:35.719104 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod148b41c5_11e6_4fcf_9bd8_34f08d15f184.slice/crio-0ee74731e647e15064fb9c4180953353790b03fffc9f76535855285c5c76e38f WatchSource:0}: Error finding container 0ee74731e647e15064fb9c4180953353790b03fffc9f76535855285c5c76e38f: Status 404 returned error can't find the container with id 0ee74731e647e15064fb9c4180953353790b03fffc9f76535855285c5c76e38f Jul 22 19:30:35.720692 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:35.720673 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 19:30:35.953929 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:35.953847 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140041791329312" event={"ID":"148b41c5-11e6-4fcf-9bd8-34f08d15f184","Type":"ContainerStarted","Data":"0ee74731e647e15064fb9c4180953353790b03fffc9f76535855285c5c76e38f"} Jul 22 19:30:36.959595 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:36.959558 2574 generic.go:356] "Generic (PLEG): container finished" podID="148b41c5-11e6-4fcf-9bd8-34f08d15f184" containerID="70ea6b9cd5094802f2ba48c49e528b2600506e272aa792b0682ace3d3af876dd" exitCode=0 Jul 22 19:30:36.960004 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:36.959648 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140041791329312" event={"ID":"148b41c5-11e6-4fcf-9bd8-34f08d15f184","Type":"ContainerDied","Data":"70ea6b9cd5094802f2ba48c49e528b2600506e272aa792b0682ace3d3af876dd"} Jul 22 19:30:38.089248 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.089223 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:38.108831 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.108808 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7789-140041791329312_148b41c5-11e6-4fcf-9bd8-34f08d15f184/test-curl-7789-140041791329312/0.log" Jul 22 19:30:38.130411 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.130379 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7789-140041791329312"] Jul 22 19:30:38.133778 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.133754 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7789-140041791329312"] Jul 22 19:30:38.185838 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.185803 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rpp6\" (UniqueName: \"kubernetes.io/projected/148b41c5-11e6-4fcf-9bd8-34f08d15f184-kube-api-access-9rpp6\") pod \"148b41c5-11e6-4fcf-9bd8-34f08d15f184\" (UID: \"148b41c5-11e6-4fcf-9bd8-34f08d15f184\") " Jul 22 19:30:38.187721 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.187699 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148b41c5-11e6-4fcf-9bd8-34f08d15f184-kube-api-access-9rpp6" (OuterVolumeSpecName: "kube-api-access-9rpp6") pod "148b41c5-11e6-4fcf-9bd8-34f08d15f184" (UID: "148b41c5-11e6-4fcf-9bd8-34f08d15f184"). InnerVolumeSpecName "kube-api-access-9rpp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:30:38.258471 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.258446 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7789-140041791329312"] Jul 22 19:30:38.258825 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.258812 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="148b41c5-11e6-4fcf-9bd8-34f08d15f184" containerName="test-curl-7789-140041791329312" Jul 22 19:30:38.258873 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.258827 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="148b41c5-11e6-4fcf-9bd8-34f08d15f184" containerName="test-curl-7789-140041791329312" Jul 22 19:30:38.258906 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.258876 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="148b41c5-11e6-4fcf-9bd8-34f08d15f184" containerName="test-curl-7789-140041791329312" Jul 22 19:30:38.261763 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.261749 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:38.265849 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.265826 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140041791329312"] Jul 22 19:30:38.287251 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.287230 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rpp6\" (UniqueName: \"kubernetes.io/projected/148b41c5-11e6-4fcf-9bd8-34f08d15f184-kube-api-access-9rpp6\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:30:38.388022 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.387990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwqhm\" (UniqueName: \"kubernetes.io/projected/3829ea5a-0e7a-4502-9b2a-59891dee64b9-kube-api-access-rwqhm\") pod \"test-curl-7789-140041791329312\" (UID: \"3829ea5a-0e7a-4502-9b2a-59891dee64b9\") " pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:38.488756 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.488728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwqhm\" (UniqueName: \"kubernetes.io/projected/3829ea5a-0e7a-4502-9b2a-59891dee64b9-kube-api-access-rwqhm\") pod \"test-curl-7789-140041791329312\" (UID: \"3829ea5a-0e7a-4502-9b2a-59891dee64b9\") " pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:38.496394 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.496373 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwqhm\" (UniqueName: \"kubernetes.io/projected/3829ea5a-0e7a-4502-9b2a-59891dee64b9-kube-api-access-rwqhm\") pod \"test-curl-7789-140041791329312\" (UID: \"3829ea5a-0e7a-4502-9b2a-59891dee64b9\") " pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:38.570912 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.570851 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:38.695513 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.695446 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140041791329312"] Jul 22 19:30:38.697560 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:30:38.697536 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3829ea5a_0e7a_4502_9b2a_59891dee64b9.slice/crio-2199adadb5f3fb9dd300943df3712092a368e448e260a37de9fd04873f021d3d WatchSource:0}: Error finding container 2199adadb5f3fb9dd300943df3712092a368e448e260a37de9fd04873f021d3d: Status 404 returned error can't find the container with id 2199adadb5f3fb9dd300943df3712092a368e448e260a37de9fd04873f021d3d Jul 22 19:30:38.969957 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.969870 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140041791329312" event={"ID":"3829ea5a-0e7a-4502-9b2a-59891dee64b9","Type":"ContainerStarted","Data":"2199adadb5f3fb9dd300943df3712092a368e448e260a37de9fd04873f021d3d"} Jul 22 19:30:38.971004 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.970967 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee74731e647e15064fb9c4180953353790b03fffc9f76535855285c5c76e38f" Jul 22 19:30:38.971126 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.971018 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:38.982156 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:38.982121 2574 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7789-140041791329312" oldPodUID="148b41c5-11e6-4fcf-9bd8-34f08d15f184" podUID="3829ea5a-0e7a-4502-9b2a-59891dee64b9" Jul 22 19:30:39.977153 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:39.977118 2574 generic.go:356] "Generic (PLEG): container finished" podID="3829ea5a-0e7a-4502-9b2a-59891dee64b9" containerID="6e17f9672c4fb707904b59a47408e2c5b49a84a126f7c9c74ece52162410195c" exitCode=0 Jul 22 19:30:39.977544 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:39.977189 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140041791329312" event={"ID":"3829ea5a-0e7a-4502-9b2a-59891dee64b9","Type":"ContainerDied","Data":"6e17f9672c4fb707904b59a47408e2c5b49a84a126f7c9c74ece52162410195c"} Jul 22 19:30:40.004756 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:40.004722 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="148b41c5-11e6-4fcf-9bd8-34f08d15f184" path="/var/lib/kubelet/pods/148b41c5-11e6-4fcf-9bd8-34f08d15f184/volumes" Jul 22 19:30:41.119604 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.119581 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:41.139072 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.139049 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7789-140041791329312_3829ea5a-0e7a-4502-9b2a-59891dee64b9/test-curl-7789-140041791329312/0.log" Jul 22 19:30:41.160455 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.160428 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7789-140041791329312"] Jul 22 19:30:41.165031 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.165010 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7789-140041791329312"] Jul 22 19:30:41.309452 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.309375 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7789-140041791330320"] Jul 22 19:30:41.309735 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.309722 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3829ea5a-0e7a-4502-9b2a-59891dee64b9" containerName="test-curl-7789-140041791329312" Jul 22 19:30:41.309776 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.309736 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3829ea5a-0e7a-4502-9b2a-59891dee64b9" containerName="test-curl-7789-140041791329312" Jul 22 19:30:41.309821 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.309811 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3829ea5a-0e7a-4502-9b2a-59891dee64b9" containerName="test-curl-7789-140041791329312" Jul 22 19:30:41.312465 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.312447 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwqhm\" (UniqueName: \"kubernetes.io/projected/3829ea5a-0e7a-4502-9b2a-59891dee64b9-kube-api-access-rwqhm\") pod \"3829ea5a-0e7a-4502-9b2a-59891dee64b9\" (UID: \"3829ea5a-0e7a-4502-9b2a-59891dee64b9\") " Jul 22 19:30:41.312557 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.312541 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:41.314884 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.314854 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3829ea5a-0e7a-4502-9b2a-59891dee64b9-kube-api-access-rwqhm" (OuterVolumeSpecName: "kube-api-access-rwqhm") pod "3829ea5a-0e7a-4502-9b2a-59891dee64b9" (UID: "3829ea5a-0e7a-4502-9b2a-59891dee64b9"). InnerVolumeSpecName "kube-api-access-rwqhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:30:41.318310 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.318286 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140041791330320"] Jul 22 19:30:41.413562 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.413528 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndgjk\" (UniqueName: \"kubernetes.io/projected/4b03a511-5aeb-44b2-b313-6485149503ff-kube-api-access-ndgjk\") pod \"test-curl-7789-140041791330320\" (UID: \"4b03a511-5aeb-44b2-b313-6485149503ff\") " pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:41.413744 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.413630 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwqhm\" (UniqueName: \"kubernetes.io/projected/3829ea5a-0e7a-4502-9b2a-59891dee64b9-kube-api-access-rwqhm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:30:41.514992 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.514931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndgjk\" (UniqueName: \"kubernetes.io/projected/4b03a511-5aeb-44b2-b313-6485149503ff-kube-api-access-ndgjk\") pod \"test-curl-7789-140041791330320\" (UID: \"4b03a511-5aeb-44b2-b313-6485149503ff\") " pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:41.523221 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.523191 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndgjk\" (UniqueName: \"kubernetes.io/projected/4b03a511-5aeb-44b2-b313-6485149503ff-kube-api-access-ndgjk\") pod \"test-curl-7789-140041791330320\" (UID: \"4b03a511-5aeb-44b2-b313-6485149503ff\") " pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:41.630050 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.629956 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:41.754718 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.754692 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140041791330320"] Jul 22 19:30:41.757006 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:30:41.756963 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b03a511_5aeb_44b2_b313_6485149503ff.slice/crio-a5132f68df7c88a097962f6ff81ae8e7479757523cab4a65dbeb1c12beaec958 WatchSource:0}: Error finding container a5132f68df7c88a097962f6ff81ae8e7479757523cab4a65dbeb1c12beaec958: Status 404 returned error can't find the container with id a5132f68df7c88a097962f6ff81ae8e7479757523cab4a65dbeb1c12beaec958 Jul 22 19:30:41.986426 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.986388 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140041791330320" event={"ID":"4b03a511-5aeb-44b2-b313-6485149503ff","Type":"ContainerStarted","Data":"a5132f68df7c88a097962f6ff81ae8e7479757523cab4a65dbeb1c12beaec958"} Jul 22 19:30:41.987537 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.987511 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:41.987656 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:41.987539 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2199adadb5f3fb9dd300943df3712092a368e448e260a37de9fd04873f021d3d" Jul 22 19:30:41.997645 ip-10-0-143-15 kubenswrapper[2574]: E0722 19:30:41.997614 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7789-140041791329312\" is forbidden: User \"system:node:ip-10-0-143-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-15.ec2.internal' and this object" podUID="3829ea5a-0e7a-4502-9b2a-59891dee64b9" pod="opendatahub/test-curl-7789-140041791329312" Jul 22 19:30:42.004410 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.004388 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3829ea5a-0e7a-4502-9b2a-59891dee64b9" path="/var/lib/kubelet/pods/3829ea5a-0e7a-4502-9b2a-59891dee64b9/volumes" Jul 22 19:30:42.400039 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.399935 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" podUID="08a9876f-a833-490c-8a28-3f62dcb1ea2b" containerName="main" containerID="cri-o://1400b808245d61cc389143045bc45175ea515e20f584e19454ff770f3a8b2a13" gracePeriod=45 Jul 22 19:30:42.654546 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.654495 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:30:42.725834 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.725803 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a9876f-a833-490c-8a28-3f62dcb1ea2b-tls-certs\") pod \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " Jul 22 19:30:42.726017 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.725846 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-dshm\") pod \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " Jul 22 19:30:42.726017 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.725996 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxcdn\" (UniqueName: \"kubernetes.io/projected/08a9876f-a833-490c-8a28-3f62dcb1ea2b-kube-api-access-wxcdn\") pod \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " Jul 22 19:30:42.726151 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.726055 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-home\") pod \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " Jul 22 19:30:42.726151 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.726097 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-tmp-dir\") pod \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " Jul 22 19:30:42.726151 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.726135 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-model-cache\") pod \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\" (UID: \"08a9876f-a833-490c-8a28-3f62dcb1ea2b\") " Jul 22 19:30:42.726388 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.726337 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-home" (OuterVolumeSpecName: "home") pod "08a9876f-a833-490c-8a28-3f62dcb1ea2b" (UID: "08a9876f-a833-490c-8a28-3f62dcb1ea2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:30:42.726388 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.726357 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08a9876f-a833-490c-8a28-3f62dcb1ea2b" (UID: "08a9876f-a833-490c-8a28-3f62dcb1ea2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:30:42.726535 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.726398 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-model-cache" (OuterVolumeSpecName: "model-cache") pod "08a9876f-a833-490c-8a28-3f62dcb1ea2b" (UID: "08a9876f-a833-490c-8a28-3f62dcb1ea2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:30:42.728109 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.728086 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-dshm" (OuterVolumeSpecName: "dshm") pod "08a9876f-a833-490c-8a28-3f62dcb1ea2b" (UID: "08a9876f-a833-490c-8a28-3f62dcb1ea2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:30:42.728198 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.728146 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a9876f-a833-490c-8a28-3f62dcb1ea2b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08a9876f-a833-490c-8a28-3f62dcb1ea2b" (UID: "08a9876f-a833-490c-8a28-3f62dcb1ea2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:30:42.728198 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.728149 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a9876f-a833-490c-8a28-3f62dcb1ea2b-kube-api-access-wxcdn" (OuterVolumeSpecName: "kube-api-access-wxcdn") pod "08a9876f-a833-490c-8a28-3f62dcb1ea2b" (UID: "08a9876f-a833-490c-8a28-3f62dcb1ea2b"). InnerVolumeSpecName "kube-api-access-wxcdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:30:42.827518 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.827461 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxcdn\" (UniqueName: \"kubernetes.io/projected/08a9876f-a833-490c-8a28-3f62dcb1ea2b-kube-api-access-wxcdn\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:30:42.827518 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.827482 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:30:42.827518 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.827491 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:30:42.827518 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.827501 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:30:42.827518 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.827509 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a9876f-a833-490c-8a28-3f62dcb1ea2b-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:30:42.827518 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.827518 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a9876f-a833-490c-8a28-3f62dcb1ea2b-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:30:42.995832 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.995801 2574 generic.go:356] "Generic (PLEG): container finished" podID="08a9876f-a833-490c-8a28-3f62dcb1ea2b" containerID="1400b808245d61cc389143045bc45175ea515e20f584e19454ff770f3a8b2a13" exitCode=0 Jul 22 19:30:42.996022 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.995846 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" Jul 22 19:30:42.996022 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.995896 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" event={"ID":"08a9876f-a833-490c-8a28-3f62dcb1ea2b","Type":"ContainerDied","Data":"1400b808245d61cc389143045bc45175ea515e20f584e19454ff770f3a8b2a13"} Jul 22 19:30:42.996022 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.995933 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb" event={"ID":"08a9876f-a833-490c-8a28-3f62dcb1ea2b","Type":"ContainerDied","Data":"04e3e4648422791702d3e2fb53fb0ca9ac091e33f1d8c0e6eb7ac9f5faad628a"} Jul 22 19:30:42.996022 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.995957 2574 scope.go:117] "RemoveContainer" containerID="1400b808245d61cc389143045bc45175ea515e20f584e19454ff770f3a8b2a13" Jul 22 19:30:42.997515 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.997491 2574 generic.go:356] "Generic (PLEG): container finished" podID="4b03a511-5aeb-44b2-b313-6485149503ff" containerID="7d8a214aa25d94359917ca84f698f5f13bf824e623a4be7e0f0dcae4f640b07c" exitCode=0 Jul 22 19:30:42.997620 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:42.997573 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140041791330320" event={"ID":"4b03a511-5aeb-44b2-b313-6485149503ff","Type":"ContainerDied","Data":"7d8a214aa25d94359917ca84f698f5f13bf824e623a4be7e0f0dcae4f640b07c"} Jul 22 19:30:43.005434 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:43.005416 2574 scope.go:117] "RemoveContainer" containerID="1400b808245d61cc389143045bc45175ea515e20f584e19454ff770f3a8b2a13" Jul 22 19:30:43.005701 ip-10-0-143-15 kubenswrapper[2574]: E0722 19:30:43.005683 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1400b808245d61cc389143045bc45175ea515e20f584e19454ff770f3a8b2a13\": container with ID starting with 1400b808245d61cc389143045bc45175ea515e20f584e19454ff770f3a8b2a13 not found: ID does not exist" containerID="1400b808245d61cc389143045bc45175ea515e20f584e19454ff770f3a8b2a13" Jul 22 19:30:43.005748 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:43.005709 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1400b808245d61cc389143045bc45175ea515e20f584e19454ff770f3a8b2a13"} err="failed to get container status \"1400b808245d61cc389143045bc45175ea515e20f584e19454ff770f3a8b2a13\": rpc error: code = NotFound desc = could not find container \"1400b808245d61cc389143045bc45175ea515e20f584e19454ff770f3a8b2a13\": container with ID starting with 1400b808245d61cc389143045bc45175ea515e20f584e19454ff770f3a8b2a13 not found: ID does not exist" Jul 22 19:30:43.025210 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:43.025186 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb"] Jul 22 19:30:43.028930 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:43.028910 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-8f2692/rate-test-model-8f2692-kserve-94c5865bc-4j2nb"] Jul 22 19:30:44.005011 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.004964 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08a9876f-a833-490c-8a28-3f62dcb1ea2b" path="/var/lib/kubelet/pods/08a9876f-a833-490c-8a28-3f62dcb1ea2b/volumes" Jul 22 19:30:44.140391 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.140369 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:44.157588 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.157558 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7789-140041791330320_4b03a511-5aeb-44b2-b313-6485149503ff/test-curl-7789-140041791330320/0.log" Jul 22 19:30:44.177593 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.177561 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7789-140041791330320"] Jul 22 19:30:44.181046 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.181016 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7789-140041791330320"] Jul 22 19:30:44.236606 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.236586 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndgjk\" (UniqueName: \"kubernetes.io/projected/4b03a511-5aeb-44b2-b313-6485149503ff-kube-api-access-ndgjk\") pod \"4b03a511-5aeb-44b2-b313-6485149503ff\" (UID: \"4b03a511-5aeb-44b2-b313-6485149503ff\") " Jul 22 19:30:44.238580 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.238557 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b03a511-5aeb-44b2-b313-6485149503ff-kube-api-access-ndgjk" (OuterVolumeSpecName: "kube-api-access-ndgjk") pod "4b03a511-5aeb-44b2-b313-6485149503ff" (UID: "4b03a511-5aeb-44b2-b313-6485149503ff"). InnerVolumeSpecName "kube-api-access-ndgjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:30:44.309328 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.309259 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7789-140041791330320"] Jul 22 19:30:44.309606 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.309595 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a9876f-a833-490c-8a28-3f62dcb1ea2b" containerName="main" Jul 22 19:30:44.309650 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.309608 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a9876f-a833-490c-8a28-3f62dcb1ea2b" containerName="main" Jul 22 19:30:44.309650 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.309622 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b03a511-5aeb-44b2-b313-6485149503ff" containerName="test-curl-7789-140041791330320" Jul 22 19:30:44.309650 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.309628 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b03a511-5aeb-44b2-b313-6485149503ff" containerName="test-curl-7789-140041791330320" Jul 22 19:30:44.309745 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.309684 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a9876f-a833-490c-8a28-3f62dcb1ea2b" containerName="main" Jul 22 19:30:44.309745 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.309696 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b03a511-5aeb-44b2-b313-6485149503ff" containerName="test-curl-7789-140041791330320" Jul 22 19:30:44.312590 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.312575 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:44.318891 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.318865 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140041791330320"] Jul 22 19:30:44.337775 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.337743 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpwzj\" (UniqueName: \"kubernetes.io/projected/6ee713cc-9875-4635-9828-7de1dd9711fd-kube-api-access-bpwzj\") pod \"test-curl-7789-140041791330320\" (UID: \"6ee713cc-9875-4635-9828-7de1dd9711fd\") " pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:44.337918 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.337900 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndgjk\" (UniqueName: \"kubernetes.io/projected/4b03a511-5aeb-44b2-b313-6485149503ff-kube-api-access-ndgjk\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:30:44.438866 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.438830 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwzj\" (UniqueName: \"kubernetes.io/projected/6ee713cc-9875-4635-9828-7de1dd9711fd-kube-api-access-bpwzj\") pod \"test-curl-7789-140041791330320\" (UID: \"6ee713cc-9875-4635-9828-7de1dd9711fd\") " pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:44.446643 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.446623 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpwzj\" (UniqueName: \"kubernetes.io/projected/6ee713cc-9875-4635-9828-7de1dd9711fd-kube-api-access-bpwzj\") pod \"test-curl-7789-140041791330320\" (UID: \"6ee713cc-9875-4635-9828-7de1dd9711fd\") " pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:44.623455 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.623367 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:44.749306 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:44.749272 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140041791330320"] Jul 22 19:30:44.751407 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:30:44.751380 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ee713cc_9875_4635_9828_7de1dd9711fd.slice/crio-5f673001363964e3bc509864950ab6a0852af16ffa4fc7c92354582266eabd02 WatchSource:0}: Error finding container 5f673001363964e3bc509864950ab6a0852af16ffa4fc7c92354582266eabd02: Status 404 returned error can't find the container with id 5f673001363964e3bc509864950ab6a0852af16ffa4fc7c92354582266eabd02 Jul 22 19:30:45.009214 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:45.009180 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:45.009214 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:45.009195 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5132f68df7c88a097962f6ff81ae8e7479757523cab4a65dbeb1c12beaec958" Jul 22 19:30:45.010295 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:45.010265 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140041791330320" event={"ID":"6ee713cc-9875-4635-9828-7de1dd9711fd","Type":"ContainerStarted","Data":"5f673001363964e3bc509864950ab6a0852af16ffa4fc7c92354582266eabd02"} Jul 22 19:30:45.021219 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:45.021198 2574 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7789-140041791330320" oldPodUID="4b03a511-5aeb-44b2-b313-6485149503ff" podUID="6ee713cc-9875-4635-9828-7de1dd9711fd" Jul 22 19:30:46.005307 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:46.005275 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b03a511-5aeb-44b2-b313-6485149503ff" path="/var/lib/kubelet/pods/4b03a511-5aeb-44b2-b313-6485149503ff/volumes" Jul 22 19:30:46.015203 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:46.015175 2574 generic.go:356] "Generic (PLEG): container finished" podID="6ee713cc-9875-4635-9828-7de1dd9711fd" containerID="2f08b5b880d73fd977a49825693040c31b5bf6aad726a027bbff5e6aefbba8a1" exitCode=0 Jul 22 19:30:46.015527 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:46.015248 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140041791330320" event={"ID":"6ee713cc-9875-4635-9828-7de1dd9711fd","Type":"ContainerDied","Data":"2f08b5b880d73fd977a49825693040c31b5bf6aad726a027bbff5e6aefbba8a1"} Jul 22 19:30:47.162285 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.162262 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:47.182634 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.182604 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7789-140041791330320_6ee713cc-9875-4635-9828-7de1dd9711fd/test-curl-7789-140041791330320/0.log" Jul 22 19:30:47.206348 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.206324 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7789-140041791330320"] Jul 22 19:30:47.210014 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.209995 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7789-140041791330320"] Jul 22 19:30:47.260098 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.260071 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpwzj\" (UniqueName: \"kubernetes.io/projected/6ee713cc-9875-4635-9828-7de1dd9711fd-kube-api-access-bpwzj\") pod \"6ee713cc-9875-4635-9828-7de1dd9711fd\" (UID: \"6ee713cc-9875-4635-9828-7de1dd9711fd\") " Jul 22 19:30:47.262182 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.262160 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee713cc-9875-4635-9828-7de1dd9711fd-kube-api-access-bpwzj" (OuterVolumeSpecName: "kube-api-access-bpwzj") pod "6ee713cc-9875-4635-9828-7de1dd9711fd" (UID: "6ee713cc-9875-4635-9828-7de1dd9711fd"). InnerVolumeSpecName "kube-api-access-bpwzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:30:47.361229 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.361169 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpwzj\" (UniqueName: \"kubernetes.io/projected/6ee713cc-9875-4635-9828-7de1dd9711fd-kube-api-access-bpwzj\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:30:47.399596 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.399564 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn"] Jul 22 19:30:47.399962 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.399947 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ee713cc-9875-4635-9828-7de1dd9711fd" containerName="test-curl-7789-140041791330320" Jul 22 19:30:47.400027 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.399966 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ee713cc-9875-4635-9828-7de1dd9711fd" containerName="test-curl-7789-140041791330320" Jul 22 19:30:47.400065 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.400053 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ee713cc-9875-4635-9828-7de1dd9711fd" containerName="test-curl-7789-140041791330320" Jul 22 19:30:47.402937 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.402920 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn" Jul 22 19:30:47.405006 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.404985 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 19:30:47.405087 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.405003 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r5ttk\"" Jul 22 19:30:47.411958 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.411939 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn"] Jul 22 19:30:47.462178 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.462145 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e428daf6-b4cc-4ddd-bdf9-d7e883b220df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn\" (UID: \"e428daf6-b4cc-4ddd-bdf9-d7e883b220df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn" Jul 22 19:30:47.563043 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.563013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e428daf6-b4cc-4ddd-bdf9-d7e883b220df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn\" (UID: \"e428daf6-b4cc-4ddd-bdf9-d7e883b220df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn" Jul 22 19:30:47.563655 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.563635 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e428daf6-b4cc-4ddd-bdf9-d7e883b220df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn\" (UID: \"e428daf6-b4cc-4ddd-bdf9-d7e883b220df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn" Jul 22 19:30:47.715947 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.715860 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn" Jul 22 19:30:47.846297 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.846271 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn"] Jul 22 19:30:47.848253 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:30:47.848221 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode428daf6_b4cc_4ddd_bdf9_d7e883b220df.slice/crio-03cce1ecfa68c74deb23de3b163f55570724652f5ab000db6cba224c67133937 WatchSource:0}: Error finding container 03cce1ecfa68c74deb23de3b163f55570724652f5ab000db6cba224c67133937: Status 404 returned error can't find the container with id 03cce1ecfa68c74deb23de3b163f55570724652f5ab000db6cba224c67133937 Jul 22 19:30:47.873710 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.873680 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:30:47.975009 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:47.974957 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:30:48.005591 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:48.005556 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ee713cc-9875-4635-9828-7de1dd9711fd" path="/var/lib/kubelet/pods/6ee713cc-9875-4635-9828-7de1dd9711fd/volumes" Jul 22 19:30:48.025420 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:48.025395 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140041791330320" Jul 22 19:30:48.025519 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:48.025455 2574 scope.go:117] "RemoveContainer" containerID="2f08b5b880d73fd977a49825693040c31b5bf6aad726a027bbff5e6aefbba8a1" Jul 22 19:30:48.026766 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:48.026743 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn" event={"ID":"e428daf6-b4cc-4ddd-bdf9-d7e883b220df","Type":"ContainerStarted","Data":"03cce1ecfa68c74deb23de3b163f55570724652f5ab000db6cba224c67133937"} Jul 22 19:30:49.034195 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:49.034161 2574 generic.go:356] "Generic (PLEG): container finished" podID="e428daf6-b4cc-4ddd-bdf9-d7e883b220df" containerID="3c1f5613265abe9c85a10df258b4ac82f6905176f940327b8f8d29c7085d6e36" exitCode=0 Jul 22 19:30:49.034553 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:49.034205 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn" event={"ID":"e428daf6-b4cc-4ddd-bdf9-d7e883b220df","Type":"ContainerDied","Data":"3c1f5613265abe9c85a10df258b4ac82f6905176f940327b8f8d29c7085d6e36"} Jul 22 19:30:50.177647 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:50.177622 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn" Jul 22 19:30:50.184686 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:50.184660 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e428daf6-b4cc-4ddd-bdf9-d7e883b220df-maas-api-service-ca\") pod \"e428daf6-b4cc-4ddd-bdf9-d7e883b220df\" (UID: \"e428daf6-b4cc-4ddd-bdf9-d7e883b220df\") " Jul 22 19:30:50.184987 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:50.184949 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e428daf6-b4cc-4ddd-bdf9-d7e883b220df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e428daf6-b4cc-4ddd-bdf9-d7e883b220df" (UID: "e428daf6-b4cc-4ddd-bdf9-d7e883b220df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:30:50.286140 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:50.286107 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e428daf6-b4cc-4ddd-bdf9-d7e883b220df-maas-api-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:30:51.049866 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:51.049831 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn" Jul 22 19:30:51.049866 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:51.049851 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn" event={"ID":"e428daf6-b4cc-4ddd-bdf9-d7e883b220df","Type":"ContainerDied","Data":"03cce1ecfa68c74deb23de3b163f55570724652f5ab000db6cba224c67133937"} Jul 22 19:30:51.050093 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:51.049904 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03cce1ecfa68c74deb23de3b163f55570724652f5ab000db6cba224c67133937" Jul 22 19:30:57.811495 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:30:57.811456 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:02.477814 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:02.477785 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:02.489093 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:02.489062 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn"] Jul 22 19:31:02.493281 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:02.493257 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4ac498-xzvwn"] Jul 22 19:31:02.998004 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:02.997951 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:04.005702 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:04.005670 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e428daf6-b4cc-4ddd-bdf9-d7e883b220df" path="/var/lib/kubelet/pods/e428daf6-b4cc-4ddd-bdf9-d7e883b220df/volumes" Jul 22 19:31:08.999599 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:08.999561 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:09.100151 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.100116 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:09.274886 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.274788 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:09.568454 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.568352 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:09.670170 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.670133 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j"] Jul 22 19:31:09.670739 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.670718 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e428daf6-b4cc-4ddd-bdf9-d7e883b220df" containerName="revoke-keys" Jul 22 19:31:09.670788 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.670744 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e428daf6-b4cc-4ddd-bdf9-d7e883b220df" containerName="revoke-keys" Jul 22 19:31:09.670839 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.670828 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e428daf6-b4cc-4ddd-bdf9-d7e883b220df" containerName="revoke-keys" Jul 22 19:31:09.675372 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.675355 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j" Jul 22 19:31:09.677746 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.677719 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 19:31:09.678430 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.678406 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r5ttk\"" Jul 22 19:31:09.680328 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.680307 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j"] Jul 22 19:31:09.750789 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.750757 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2f140b-c032-46c9-a152-c6fd6159bd98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j\" (UID: \"5d2f140b-c032-46c9-a152-c6fd6159bd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j" Jul 22 19:31:09.851994 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.851895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2f140b-c032-46c9-a152-c6fd6159bd98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j\" (UID: \"5d2f140b-c032-46c9-a152-c6fd6159bd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j" Jul 22 19:31:09.852508 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.852490 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2f140b-c032-46c9-a152-c6fd6159bd98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j\" (UID: \"5d2f140b-c032-46c9-a152-c6fd6159bd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j" Jul 22 19:31:09.987787 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:09.987746 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j" Jul 22 19:31:10.117197 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:10.117137 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j"] Jul 22 19:31:10.119852 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:31:10.119825 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2f140b_c032_46c9_a152_c6fd6159bd98.slice/crio-ddf2704140e12e35d9c62f99f244ead155cace87e77d63291d12c52fb4c31e19 WatchSource:0}: Error finding container ddf2704140e12e35d9c62f99f244ead155cace87e77d63291d12c52fb4c31e19: Status 404 returned error can't find the container with id ddf2704140e12e35d9c62f99f244ead155cace87e77d63291d12c52fb4c31e19 Jul 22 19:31:10.140406 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:10.140351 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j" event={"ID":"5d2f140b-c032-46c9-a152-c6fd6159bd98","Type":"ContainerStarted","Data":"ddf2704140e12e35d9c62f99f244ead155cace87e77d63291d12c52fb4c31e19"} Jul 22 19:31:10.214934 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:10.214900 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:11.060825 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:11.060789 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:11.146198 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:11.146114 2574 generic.go:356] "Generic (PLEG): container finished" podID="5d2f140b-c032-46c9-a152-c6fd6159bd98" containerID="a18e1c0b294ffc7ebb57a61d99f7b6713016735649a4787c001d6759a6f7a8b7" exitCode=0 Jul 22 19:31:11.146198 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:11.146140 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j" event={"ID":"5d2f140b-c032-46c9-a152-c6fd6159bd98","Type":"ContainerDied","Data":"a18e1c0b294ffc7ebb57a61d99f7b6713016735649a4787c001d6759a6f7a8b7"} Jul 22 19:31:12.280077 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:12.280053 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j" Jul 22 19:31:12.376284 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:12.376255 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2f140b-c032-46c9-a152-c6fd6159bd98-maas-api-service-ca\") pod \"5d2f140b-c032-46c9-a152-c6fd6159bd98\" (UID: \"5d2f140b-c032-46c9-a152-c6fd6159bd98\") " Jul 22 19:31:12.376600 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:12.376579 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2f140b-c032-46c9-a152-c6fd6159bd98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d2f140b-c032-46c9-a152-c6fd6159bd98" (UID: "5d2f140b-c032-46c9-a152-c6fd6159bd98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:31:12.477579 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:12.477554 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2f140b-c032-46c9-a152-c6fd6159bd98-maas-api-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\"" Jul 22 19:31:13.156591 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:13.156562 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j" Jul 22 19:31:13.156761 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:13.156602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j" event={"ID":"5d2f140b-c032-46c9-a152-c6fd6159bd98","Type":"ContainerDied","Data":"ddf2704140e12e35d9c62f99f244ead155cace87e77d63291d12c52fb4c31e19"} Jul 22 19:31:13.156761 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:13.156645 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddf2704140e12e35d9c62f99f244ead155cace87e77d63291d12c52fb4c31e19" Jul 22 19:31:20.099258 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:20.099222 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:24.742546 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:24.742505 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j"] Jul 22 19:31:24.743095 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:24.742744 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:24.744877 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:24.744856 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8f2692-zfx7j"] Jul 22 19:31:25.144830 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:25.144734 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:26.004753 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:26.004714 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d2f140b-c032-46c9-a152-c6fd6159bd98" path="/var/lib/kubelet/pods/5d2f140b-c032-46c9-a152-c6fd6159bd98/volumes" Jul 22 19:31:30.839835 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:30.839802 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:30.900057 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:30.900019 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:31.140451 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:31.140335 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:31.407650 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:31.407554 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:38.909582 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:38.909548 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-64rcl_c37545e7-7734-43c0-bfd4-714ed76c147f/manager/0.log" Jul 22 19:31:39.724132 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:39.724101 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f9bb58876-lrjlc_a02f7981-bbd1-41d3-bbcd-584bab80e60b/copy-manifests/0.log" Jul 22 19:31:39.730884 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:39.730862 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f9bb58876-lrjlc_a02f7981-bbd1-41d3-bbcd-584bab80e60b/manager/0.log" Jul 22 19:31:43.248130 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:43.248089 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-j98sb_38f59f5f-c160-495c-bedf-65b1c284b7db/manager/0.log" Jul 22 19:31:43.998434 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:43.998395 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779ddf9494-gj7dh_2e4bf375-301f-4d5d-b0e8-154256b89782/kube-auth-proxy/0.log" Jul 22 19:31:44.382339 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:44.382257 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-h4rb8_ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a/payload-pre-processing/0.log" Jul 22 19:31:45.014009 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:45.013951 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-vkjc8_76c042e5-202b-4e4f-8fa3-ed9e18fffc9f/main/0.log" Jul 22 19:31:45.498993 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:45.498935 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-lpzhd_f10818ee-3735-4efb-ba1f-5ea97839b358/main/0.log" Jul 22 19:31:45.626590 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:45.626534 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-nvqc6_427fa00a-700d-4abb-ac22-037f50135707/main/0.log" Jul 22 19:31:51.090329 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:51.090282 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:51.416519 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:51.416390 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:51.472369 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:51.472332 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:52.295756 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:52.295713 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:52.361280 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:52.361245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" limit=1024 Jul 22 19:31:55.298158 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:55.298121 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9xjws_0f37a932-854e-4799-894e-db1e2b8a27d3/global-pull-secret-syncer/0.log" Jul 22 19:31:55.390585 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:55.390556 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8n5kx_64894f60-a8ee-4da7-a605-e109bb3a84aa/konnectivity-agent/0.log" Jul 22 19:31:55.537400 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:31:55.537367 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-15.ec2.internal_5982cdd9aa07454c3773d04a38a8a229/haproxy/0.log" Jul 22 19:32:00.956539 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:00.956447 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-j98sb_38f59f5f-c160-495c-bedf-65b1c284b7db/manager/0.log" Jul 22 19:32:02.542550 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:02.542504 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jtlfw_9a95b2ad-2f10-4864-9402-7ca7c1a930bd/kube-state-metrics/0.log" Jul 22 19:32:02.557220 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:02.557155 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jtlfw_9a95b2ad-2f10-4864-9402-7ca7c1a930bd/kube-rbac-proxy-main/0.log" Jul 22 19:32:02.575509 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:02.575486 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jtlfw_9a95b2ad-2f10-4864-9402-7ca7c1a930bd/kube-rbac-proxy-self/0.log" Jul 22 19:32:02.657311 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:02.657288 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2pth_070aea45-a91b-4d2b-8883-11c40e0086ce/node-exporter/0.log" Jul 22 19:32:02.678554 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:02.678530 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2pth_070aea45-a91b-4d2b-8883-11c40e0086ce/kube-rbac-proxy/0.log" Jul 22 19:32:02.697487 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:02.697461 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2pth_070aea45-a91b-4d2b-8883-11c40e0086ce/init-textfile/0.log" Jul 22 19:32:02.983523 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:02.983497 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5/prometheus/0.log" Jul 22 19:32:03.006548 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:03.006516 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5/config-reloader/0.log" Jul 22 19:32:03.026217 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:03.026189 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5/thanos-sidecar/0.log" Jul 22 19:32:03.050207 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:03.050179 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5/kube-rbac-proxy-web/0.log" Jul 22 19:32:03.070875 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:03.070511 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5/kube-rbac-proxy/0.log" Jul 22 19:32:03.090995 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:03.090947 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5/kube-rbac-proxy-thanos/0.log" Jul 22 19:32:03.112713 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:03.112668 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7dcf2bbc-8811-4e3b-bb53-ad8dd34213c5/init-config-reloader/0.log" Jul 22 19:32:03.140873 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:03.140803 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-hxjw6_890548e6-01a9-4d90-a6b3-781b15f8ec79/prometheus-operator/0.log" Jul 22 19:32:03.167110 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:03.167087 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-hxjw6_890548e6-01a9-4d90-a6b3-781b15f8ec79/kube-rbac-proxy/0.log" Jul 22 19:32:03.200159 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:03.200129 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-sznbc_87e51017-f0f2-43c4-ab04-9f98e893b8e8/prometheus-operator-admission-webhook/0.log" Jul 22 19:32:04.715027 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:04.714996 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-78xp8_b117ac7f-9bc8-46e2-a049-fdc145a52523/networking-console-plugin/0.log" Jul 22 19:32:06.334752 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.334714 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh"] Jul 22 19:32:06.335208 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.335062 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2f140b-c032-46c9-a152-c6fd6159bd98" containerName="revoke-keys" Jul 22 19:32:06.335208 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.335074 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2f140b-c032-46c9-a152-c6fd6159bd98" containerName="revoke-keys" Jul 22 19:32:06.335208 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.335142 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2f140b-c032-46c9-a152-c6fd6159bd98" containerName="revoke-keys" Jul 22 19:32:06.338303 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.338281 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.340548 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.340532 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4nfc7\"/\"kube-root-ca.crt\"" Jul 22 19:32:06.340644 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.340593 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4nfc7\"/\"openshift-service-ca.crt\"" Jul 22 19:32:06.340725 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.340706 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4nfc7\"/\"default-dockercfg-dxrwg\"" Jul 22 19:32:06.348068 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.348049 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh"] Jul 22 19:32:06.435851 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.435824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98276f95-be75-450e-bde5-abb20c92c87f-podres\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.436017 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.435884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98276f95-be75-450e-bde5-abb20c92c87f-proc\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.436017 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.435951 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98276f95-be75-450e-bde5-abb20c92c87f-lib-modules\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.436017 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.436003 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98276f95-be75-450e-bde5-abb20c92c87f-sys\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.436136 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.436032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsdlg\" (UniqueName: \"kubernetes.io/projected/98276f95-be75-450e-bde5-abb20c92c87f-kube-api-access-hsdlg\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.537114 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.537080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98276f95-be75-450e-bde5-abb20c92c87f-sys\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.537292 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.537126 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsdlg\" (UniqueName: \"kubernetes.io/projected/98276f95-be75-450e-bde5-abb20c92c87f-kube-api-access-hsdlg\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.537292 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.537151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98276f95-be75-450e-bde5-abb20c92c87f-podres\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.537292 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.537214 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98276f95-be75-450e-bde5-abb20c92c87f-proc\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.537292 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.537223 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98276f95-be75-450e-bde5-abb20c92c87f-sys\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.537292 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.537254 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98276f95-be75-450e-bde5-abb20c92c87f-lib-modules\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.537292 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.537293 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98276f95-be75-450e-bde5-abb20c92c87f-proc\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.537519 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.537348 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98276f95-be75-450e-bde5-abb20c92c87f-podres\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.537519 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.537377 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98276f95-be75-450e-bde5-abb20c92c87f-lib-modules\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.544804 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.544777 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsdlg\" (UniqueName: \"kubernetes.io/projected/98276f95-be75-450e-bde5-abb20c92c87f-kube-api-access-hsdlg\") pod \"perf-node-gather-daemonset-7lssh\" (UID: \"98276f95-be75-450e-bde5-abb20c92c87f\") " pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.648424 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.648349 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:06.786393 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:06.786368 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh"] Jul 22 19:32:06.788864 ip-10-0-143-15 kubenswrapper[2574]: W0722 19:32:06.788834 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod98276f95_be75_450e_bde5_abb20c92c87f.slice/crio-dc916dc82fc50c52a8edc876ddabb1d7696463cae7a5b3b8c54d742ae995ba53 WatchSource:0}: Error finding container dc916dc82fc50c52a8edc876ddabb1d7696463cae7a5b3b8c54d742ae995ba53: Status 404 returned error can't find the container with id dc916dc82fc50c52a8edc876ddabb1d7696463cae7a5b3b8c54d742ae995ba53 Jul 22 19:32:07.419343 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:07.419306 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" event={"ID":"98276f95-be75-450e-bde5-abb20c92c87f","Type":"ContainerStarted","Data":"92d1c951ec273467f48498f52ec0d4aaea10b5f5a4f2e1f3982d6a0ba98ec481"} Jul 22 19:32:07.419749 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:07.419362 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" event={"ID":"98276f95-be75-450e-bde5-abb20c92c87f","Type":"ContainerStarted","Data":"dc916dc82fc50c52a8edc876ddabb1d7696463cae7a5b3b8c54d742ae995ba53"} Jul 22 19:32:07.419749 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:07.419425 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:07.434925 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:07.434882 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" podStartSLOduration=1.4348670860000001 podStartE2EDuration="1.434867086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 19:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:32:07.434076867 +0000 UTC m=+3137.881472099" watchObservedRunningTime="2026-07-22 19:32:07.434867086 +0000 UTC m=+3137.882262320" Jul 22 19:32:07.721865 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:07.721784 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x262w_42cc3535-6546-4f2d-a502-9f3a97a63eaa/dns/0.log" Jul 22 19:32:07.741565 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:07.741538 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x262w_42cc3535-6546-4f2d-a502-9f3a97a63eaa/kube-rbac-proxy/0.log" Jul 22 19:32:07.810793 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:07.810770 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7kgx_daf45312-be61-4ecc-bf68-ee249f228c2b/dns-node-resolver/0.log" Jul 22 19:32:08.390377 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:08.390348 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gg7vx_4e3a6ef6-5c0f-4077-9228-9fbd12b1c5f4/node-ca/0.log" Jul 22 19:32:09.429796 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:09.429770 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779ddf9494-gj7dh_2e4bf375-301f-4d5d-b0e8-154256b89782/kube-auth-proxy/0.log" Jul 22 19:32:09.579955 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:09.579927 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-h4rb8_ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a/payload-pre-processing/0.log" Jul 22 19:32:10.308110 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:10.308085 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8ph6g_7fe1adde-5816-463f-aa6a-7769d7569b7f/serve-healthcheck-canary/0.log" Jul 22 19:32:10.873128 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:10.873096 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hgnhk_c44ab18a-bae2-4048-a505-bb14abbe90b7/kube-rbac-proxy/0.log" Jul 22 19:32:10.891244 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:10.891220 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hgnhk_c44ab18a-bae2-4048-a505-bb14abbe90b7/exporter/0.log" Jul 22 19:32:10.910792 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:10.910768 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hgnhk_c44ab18a-bae2-4048-a505-bb14abbe90b7/extractor/0.log" Jul 22 19:32:12.969628 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:12.969558 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-64rcl_c37545e7-7734-43c0-bfd4-714ed76c147f/manager/0.log" Jul 22 19:32:13.433732 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:13.433704 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4nfc7/perf-node-gather-daemonset-7lssh" Jul 22 19:32:13.750601 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:13.750573 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f9bb58876-lrjlc_a02f7981-bbd1-41d3-bbcd-584bab80e60b/manager/0.log" Jul 22 19:32:13.767260 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:13.767236 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f9bb58876-lrjlc_a02f7981-bbd1-41d3-bbcd-584bab80e60b/copy-manifests/0.log" Jul 22 19:32:15.207518 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:15.207443 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b444cc6d8-rqclw_c576f0eb-eb3e-439f-9146-87fd146e8911/manager/0.log" Jul 22 19:32:15.696019 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:15.695988 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779ddf9494-gj7dh_2e4bf375-301f-4d5d-b0e8-154256b89782/kube-auth-proxy/0.log" Jul 22 19:32:15.819489 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:15.819462 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-h4rb8_ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a/payload-pre-processing/0.log" Jul 22 19:32:17.169015 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:17.168989 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779ddf9494-gj7dh_2e4bf375-301f-4d5d-b0e8-154256b89782/kube-auth-proxy/0.log" Jul 22 19:32:17.299490 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:17.299460 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-h4rb8_ab7f5c1b-1c3b-4a9d-9514-b6616aa94b2a/payload-pre-processing/0.log" Jul 22 19:32:20.308709 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:20.308682 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-vhvgj_16d275cd-541e-4cfa-80ab-68c617f381bc/migrator/0.log" Jul 22 19:32:20.327425 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:20.327399 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-vhvgj_16d275cd-541e-4cfa-80ab-68c617f381bc/graceful-termination/0.log" Jul 22 19:32:21.603272 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:21.603246 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ztgb_73a193b7-d3c9-451c-af5c-fe08756dba3f/kube-multus-additional-cni-plugins/0.log" Jul 22 19:32:21.623030 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:21.622999 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ztgb_73a193b7-d3c9-451c-af5c-fe08756dba3f/egress-router-binary-copy/0.log" Jul 22 19:32:21.642080 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:21.642058 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ztgb_73a193b7-d3c9-451c-af5c-fe08756dba3f/cni-plugins/0.log" Jul 22 19:32:21.659708 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:21.659690 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ztgb_73a193b7-d3c9-451c-af5c-fe08756dba3f/bond-cni-plugin/0.log" Jul 22 19:32:21.677856 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:21.677833 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ztgb_73a193b7-d3c9-451c-af5c-fe08756dba3f/routeoverride-cni/0.log" Jul 22 19:32:21.695295 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:21.695273 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ztgb_73a193b7-d3c9-451c-af5c-fe08756dba3f/whereabouts-cni-bincopy/0.log" Jul 22 19:32:21.712904 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:21.712884 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ztgb_73a193b7-d3c9-451c-af5c-fe08756dba3f/whereabouts-cni/0.log" Jul 22 19:32:22.110791 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:22.110753 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zt4fh_e953389b-d0c5-4bc7-907a-1687430ff7ba/kube-multus/0.log" Jul 22 19:32:22.130197 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:22.130165 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lc7qt_d27e81e3-a9d6-4c4b-b138-3a4394204eb3/network-metrics-daemon/0.log" Jul 22 19:32:22.145500 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:22.145476 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lc7qt_d27e81e3-a9d6-4c4b-b138-3a4394204eb3/kube-rbac-proxy/0.log" Jul 22 19:32:23.209958 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:23.209906 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s8l6_a87719eb-a436-4abc-99f3-db0c8db21201/ovn-controller/0.log" Jul 22 19:32:23.224586 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:23.224561 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s8l6_a87719eb-a436-4abc-99f3-db0c8db21201/ovn-acl-logging/0.log" Jul 22 19:32:23.253303 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:23.253276 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s8l6_a87719eb-a436-4abc-99f3-db0c8db21201/kube-rbac-proxy-node/0.log" Jul 22 19:32:23.275441 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:23.275414 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s8l6_a87719eb-a436-4abc-99f3-db0c8db21201/kube-rbac-proxy-ovn-metrics/0.log" Jul 22 19:32:23.290157 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:23.290138 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s8l6_a87719eb-a436-4abc-99f3-db0c8db21201/northd/0.log" Jul 22 19:32:23.307580 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:23.307562 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s8l6_a87719eb-a436-4abc-99f3-db0c8db21201/nbdb/0.log" Jul 22 19:32:23.325025 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:23.324987 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s8l6_a87719eb-a436-4abc-99f3-db0c8db21201/sbdb/0.log" Jul 22 19:32:23.402875 ip-10-0-143-15 kubenswrapper[2574]: I0722 19:32:23.402851 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s8l6_a87719eb-a436-4abc-99f3-db0c8db21201/ovnkube-controller/0.log"