Jul 22 11:47:33.891885 ip-10-0-130-190 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 11:47:33.891898 ip-10-0-130-190 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 11:47:33.891908 ip-10-0-130-190 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 11:47:33.892251 ip-10-0-130-190 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 11:47:43.950005 ip-10-0-130-190 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 11:47:43.950028 ip-10-0-130-190 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8f94c5bca945419da20a2928a22ad249 -- Jul 22 11:49:36.051190 ip-10-0-130-190 systemd[1]: Starting Kubernetes Kubelet... Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.428896 2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434061 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434075 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434078 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434082 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434085 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434088 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434090 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:49:36.445817 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434093 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434096 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434098 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434101 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434104 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434107 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434110 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434113 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434115 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434118 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434121 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434124 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434127 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434133 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434135 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434138 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434140 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434143 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434146 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434149 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:49:36.459767 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434151 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434154 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434157 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434161 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434164 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434166 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434170 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434173 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434176 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434179 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434181 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434184 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434186 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434189 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434191 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434194 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434196 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434199 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434202 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:49:36.463820 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434206 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434210 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434214 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434217 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434220 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434222 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434225 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434228 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434231 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434234 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434236 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434239 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434241 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434244 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434246 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434250 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434252 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434255 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434257 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434260 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:49:36.478228 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434263 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434265 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434268 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434270 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434274 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434276 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434278 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434281 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434284 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434286 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434289 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434292 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434296 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434299 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434302 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434304 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434307 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434310 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434312 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434315 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:49:36.482919 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434317 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434331 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434335 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434337 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434340 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434343 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434345 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434348 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434351 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434354 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434357 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434359 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.434362 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435075 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435081 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435084 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435086 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435089 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435092 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435094 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:49:37.576850 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435097 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435100 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435105 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435109 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435112 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435115 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435118 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435121 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435125 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435129 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435132 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435134 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435137 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435139 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435142 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435144 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435147 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435150 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435153 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:49:37.579340 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435156 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435158 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435161 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435164 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435166 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435169 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435172 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435175 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435177 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435180 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435183 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435185 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435189 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435192 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435195 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435198 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435201 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435204 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435207 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:49:37.614383 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435209 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:49:37.584655 ip-10-0-130-190 systemd[1]: Started Kubernetes Kubelet. Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435212 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435215 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435218 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435220 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435223 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435226 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435228 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435231 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435233 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435236 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435239 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435242 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435244 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435247 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435249 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435253 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435255 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435258 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435260 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:49:37.619587 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435263 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435265 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435268 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435271 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435273 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435277 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435279 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435282 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435285 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435287 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435290 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435292 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435296 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435298 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435301 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435303 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435306 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435308 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435311 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435313 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:49:37.624596 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435316 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435318 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435336 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435339 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435342 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435345 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435347 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435350 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435353 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435356 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435358 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435361 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435363 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435366 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435764 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435768 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435771 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435776 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435779 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:49:37.636722 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435782 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435785 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435788 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435791 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435794 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435798 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435801 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435803 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435806 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435808 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435811 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435814 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435816 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435819 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435821 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435826 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435829 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435833 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435836 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435838 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:49:37.637621 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435841 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435844 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435846 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435849 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435851 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435853 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435856 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435858 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435861 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435863 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435866 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435868 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435870 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435873 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435876 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435878 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435881 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435883 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435886 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435889 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:49:37.638454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435892 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435894 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435897 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435899 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435902 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435904 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435908 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435913 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435916 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435920 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435923 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435925 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435928 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435930 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435933 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435935 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435938 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435941 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435944 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:49:37.640899 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435947 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435950 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435952 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435955 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435957 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435960 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435962 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435965 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435967 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435970 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435973 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435976 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435979 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435982 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435984 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435987 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435989 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435993 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435995 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.435998 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:49:37.641721 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436004 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436006 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436011 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436014 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436016 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436019 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436021 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436024 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436026 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436029 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436032 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436036 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436039 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436042 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436044 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436123 2555 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436130 2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436137 2555 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436141 2555 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436145 2555 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436149 2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 11:49:37.645378 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436153 2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436157 2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436161 2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436163 2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436168 2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436171 2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436174 2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436178 2555 flags.go:64] FLAG: --cgroup-root="" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436181 2555 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436184 2555 flags.go:64] FLAG: --client-ca-file="" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436187 2555 flags.go:64] FLAG: --cloud-provider="external" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436191 2555 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436197 2555 flags.go:64] FLAG: --cluster-domain="" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436202 2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436205 2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436208 2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436212 2555 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436216 2555 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436219 2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436222 2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436225 2555 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436228 2555 flags.go:64] FLAG: --contention-profiling="false" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436231 2555 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436234 2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 11:49:37.646209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436237 2555 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436240 2555 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436245 2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436248 2555 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436251 2555 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436253 2555 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436256 2555 flags.go:64] FLAG: --enable-server="true" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436259 2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436263 2555 flags.go:64] FLAG: --event-burst="100" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436266 2555 flags.go:64] FLAG: --event-qps="50" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436269 2555 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436272 2555 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436275 2555 flags.go:64] FLAG: --eviction-hard="" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436280 2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436283 2555 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436286 2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436288 2555 flags.go:64] FLAG: --eviction-soft="" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436291 2555 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436294 2555 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436298 2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436301 2555 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436306 2555 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436309 2555 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436312 2555 flags.go:64] FLAG: --feature-gates="" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436315 2555 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 11:49:37.649470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436318 2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436336 2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436339 2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436342 2555 flags.go:64] FLAG: --healthz-port="10248" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436346 2555 flags.go:64] FLAG: --help="false" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436350 2555 flags.go:64] FLAG: --hostname-override="ip-10-0-130-190.ec2.internal" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436353 2555 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436356 2555 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436359 2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436363 2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436366 2555 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436369 2555 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436372 2555 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436375 2555 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436378 2555 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436381 2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436384 2555 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436387 2555 flags.go:64] FLAG: --kube-reserved="" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436390 2555 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436392 2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436396 2555 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436399 2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436402 2555 flags.go:64] FLAG: --lock-file="" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436405 2555 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 11:49:37.650311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436407 2555 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436410 2555 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436417 2555 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436420 2555 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436424 2555 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436427 2555 flags.go:64] FLAG: --logging-format="text" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436430 2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436433 2555 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436436 2555 flags.go:64] FLAG: --manifest-url="" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436438 2555 flags.go:64] FLAG: --manifest-url-header="" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436443 2555 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436446 2555 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436451 2555 flags.go:64] FLAG: --max-pods="110" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436454 2555 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436457 2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436460 2555 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436462 2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436465 2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436468 2555 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436471 2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436478 2555 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436481 2555 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436484 2555 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 11:49:37.755341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436487 2555 flags.go:64] FLAG: --pod-cidr="" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436490 2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436495 2555 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436498 2555 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436501 2555 flags.go:64] FLAG: --pods-per-core="0" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436504 2555 flags.go:64] FLAG: --port="10250" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436507 2555 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436511 2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ec1dbba78a54a821" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436514 2555 flags.go:64] FLAG: --qos-reserved="" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436517 2555 flags.go:64] FLAG: --read-only-port="10255" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436520 2555 flags.go:64] FLAG: --register-node="true" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436523 2555 flags.go:64] FLAG: --register-with-taints="" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436528 2555 flags.go:64] FLAG: --registry-burst="10" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436533 2555 flags.go:64] FLAG: --registry-qps="5" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436536 2555 flags.go:64] FLAG: --reserved-cpus="" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436538 2555 flags.go:64] FLAG: --reserved-memory="" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436542 2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436546 2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436549 2555 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436551 2555 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436554 2555 flags.go:64] FLAG: --runonce="false" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436557 2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436560 2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436563 2555 flags.go:64] FLAG: --seccomp-default="false" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436566 2555 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 11:49:37.756569 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436569 2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436572 2555 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436575 2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436578 2555 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436581 2555 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436583 2555 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436586 2555 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436589 2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436592 2555 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436595 2555 flags.go:64] FLAG: --system-cgroups="" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436598 2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436603 2555 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436606 2555 flags.go:64] FLAG: --tls-cert-file="" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436609 2555 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436614 2555 flags.go:64] FLAG: --tls-min-version="" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436617 2555 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436620 2555 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436623 2555 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436626 2555 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436630 2555 flags.go:64] FLAG: --v="2" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436635 2555 flags.go:64] FLAG: --version="false" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436639 2555 flags.go:64] FLAG: --vmodule="" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436643 2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.436646 2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 11:49:37.757348 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436759 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436762 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436765 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436767 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436770 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436773 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436775 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436778 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436781 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436783 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436786 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436789 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436792 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436795 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436797 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436801 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436805 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436808 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436811 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:49:37.759378 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436814 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436816 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436819 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436822 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436824 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436827 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436830 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436832 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436836 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436840 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436842 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436845 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436848 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436850 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436853 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436856 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436858 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436861 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436863 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436866 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:49:37.760219 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436869 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436871 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436874 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436877 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436879 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436882 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436885 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436887 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436891 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436895 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436897 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436900 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436902 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436905 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436908 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436910 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436913 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436915 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436918 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:49:37.761403 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436920 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436926 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436931 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436934 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436937 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436940 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436943 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436946 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436949 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436951 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436954 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436956 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436959 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436961 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436964 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436966 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436969 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436972 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436974 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436977 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:49:37.762205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436979 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436982 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436984 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436987 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436989 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436992 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436995 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.436997 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.437000 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.437003 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.437005 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.437008 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.437010 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.437015 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.437018 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.437021 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.437024 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.437026 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.437029 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.437031 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:49:37.763423 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.437034 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.437040 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.444242 2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.444260 2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.445015 2555 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.445027 2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445089 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445095 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445098 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445101 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445104 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445108 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445110 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445113 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445116 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:49:37.765465 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445119 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445122 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445125 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445127 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445130 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445132 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445135 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445138 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445140 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445143 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445145 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445148 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445151 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445153 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445156 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445158 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445162 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445166 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445169 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445172 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:49:37.765878 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445176 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445178 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445181 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445183 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445186 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445188 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445191 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445194 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445197 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445199 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445202 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445205 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445207 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445210 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445213 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445215 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445218 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445220 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445223 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445226 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:49:37.766376 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445229 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445231 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445234 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445236 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445239 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445241 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445244 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445247 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445249 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445252 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445254 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445256 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445260 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445263 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445266 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445269 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445271 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445274 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445277 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445279 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:49:37.766862 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445282 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445286 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445290 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445293 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445296 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445299 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445302 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445307 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445310 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445313 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445317 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445336 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445340 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445343 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445345 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445348 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445351 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445354 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445357 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:49:37.767349 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445359 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445362 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445365 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445367 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445370 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445373 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445375 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445379 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445381 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445384 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445386 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.445392 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445506 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445512 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445516 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:49:37.767804 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445520 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445522 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445525 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445528 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445531 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445535 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445537 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445540 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445544 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445548 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445551 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445554 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445557 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445560 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445562 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445566 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445569 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445572 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445575 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:49:37.768187 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445578 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445581 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445583 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445586 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445588 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445591 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445594 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445597 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445599 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445602 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445604 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445607 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445613 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445615 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445618 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445620 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445623 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445625 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445629 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445632 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:49:37.768648 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445634 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445637 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445639 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445642 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445644 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445647 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445649 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445652 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445654 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445657 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445660 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445662 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445664 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445667 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445670 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445672 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445674 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445677 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445679 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445682 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:49:37.769186 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445685 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445687 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445690 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445692 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445696 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445699 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445701 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445704 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445706 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445709 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445713 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445716 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445718 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445721 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445723 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445726 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445728 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445731 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445733 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445736 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:49:37.769656 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445747 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445751 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445755 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445759 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445762 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445764 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445767 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445769 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445772 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445774 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445777 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445780 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445783 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445786 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445789 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445791 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:49:37.770142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:36.445795 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:49:37.770520 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.445800 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 11:49:37.770520 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.445927 2555 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 11:49:37.770520 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.448314 2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 11:49:37.770520 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.449798 2555 server.go:1019] "Starting client certificate rotation" Jul 22 11:49:37.770520 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.449905 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 11:49:37.770520 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.449948 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 11:49:37.770520 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.452236 2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 11:49:37.770520 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.453657 2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 11:49:37.770520 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.465500 2555 log.go:25] "Validated CRI v1 runtime API" Jul 22 11:49:37.770520 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.467827 2555 log.go:25] "Validated CRI v1 image API" Jul 22 11:49:37.770520 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.469344 2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 11:49:37.770520 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.472248 2555 fs.go:135] Filesystem UUIDs: map[11277160-248f-4945-87f6-5f2c6e685c57:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f15a1f54-5379-49b5-af5d-8c74cb491571:/dev/nvme0n1p4] Jul 22 11:49:37.770821 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.472265 2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 11:49:37.770854 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.477425 2555 manager.go:216] Machine: {Timestamp:2026-07-22 11:49:36.476217783 +0000 UTC m=+0.300213425 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099607 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec245e6041d3a06e3fdc61783b5b2fd2 SystemUUID:ec245e60-41d3-a06e-3fdc-61783b5b2fd2 BootID:8f94c5bc-a945-419d-a20a-2928a22ad249 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5d:99:e6:5d:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5d:99:e6:5d:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:51:d6:81:97:5d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 11:49:37.770854 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.477526 2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 11:49:37.770854 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.477617 2555 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 11:49:37.770854 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.480347 2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 11:49:37.770854 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.480389 2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-190.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 11:49:37.770854 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.481755 2555 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 11:49:37.770854 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.481770 2555 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 11:49:37.770854 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.481780 2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.482112 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:36.482394 2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.558634 2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.563015 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.563178 2555 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.563204 2555 kubelet.go:486] "Attempting to sync node with API server" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.563214 2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.563229 2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.563239 2555 kubelet.go:398] "Adding apiserver pod source" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.563248 2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.563579 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.563595 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.569385 2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.578088 2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.578145 2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579317 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579331 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579341 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579350 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579362 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579371 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579379 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579385 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579398 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579404 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579413 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579419 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579951 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.579959 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:37.581482 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:37.581718 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.583305 2555 server.go:1268] "Started kubelet" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.588382 2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.588560 2555 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.583380 2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.589728 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ftnqm" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.590590 2555 server.go:310] "Adding debug handlers to kubelet server" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.590994 2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 11:49:37.771082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.592291 2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-190.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:37.594271 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-190.ec2.internal.18c49a45f40a1e93 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-190.ec2.internal,UID:ip-10-0-130-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-190.ec2.internal,},FirstTimestamp:2026-07-22 11:49:37.583283859 +0000 UTC m=+0.332495451,LastTimestamp:2026-07-22 11:49:37.583283859 +0000 UTC m=+0.332495451,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-190.ec2.internal,}" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:37.596655 2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.598702 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ftnqm" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.599923 2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.600325 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:37.600674 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.604341 2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:37.604344 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.604357 2555 factory.go:55] Registering systemd factory Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.604378 2555 factory.go:223] Registration of the systemd container factory successfully Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.604589 2555 factory.go:153] Registering CRI-O factory Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.604604 2555 factory.go:223] Registration of the crio container factory successfully Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.604629 2555 factory.go:103] Registering Raw factory Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.604644 2555 manager.go:1196] Started watching for new ooms in manager Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.604660 2555 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.604677 2555 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.604661 2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.604929 2555 reconstruct.go:97] "Volume reconstruction finished" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.604938 2555 reconciler.go:29] "Reconciler: start to sync state" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.605122 2555 manager.go:349] Starting recovery of all containers Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.616390 2555 manager.go:354] Recovery completed Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.622195 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.623970 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.624001 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.624012 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:37.624693 2555 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-190.ec2.internal\" not found" node="ip-10-0-130-190.ec2.internal" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.624700 2555 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.624720 2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.624729 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.624740 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.626766 2555 policy_none.go:49] "None policy: Start" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.626779 2555 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.626789 2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.628732 2555 policy_none.go:47] "Start" Jul 22 11:49:37.772040 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.630669 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:49:37.772913 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.630699 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:49:37.772913 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.630712 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:49:37.847437 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:37.847219 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:37.876692 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.876676 2555 manager.go:337] "Starting Device Plugin manager" Jul 22 11:49:37.876769 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:37.876711 2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 11:49:37.876769 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.876725 2555 server.go:87] "Starting device plugin registration server" Jul 22 11:49:37.877034 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.877021 2555 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 11:49:37.877076 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.877037 2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 11:49:37.877155 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.877137 2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 11:49:37.877230 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.877221 2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 11:49:37.877230 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.877229 2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 11:49:37.877920 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:37.877899 2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 11:49:37.878007 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:37.877958 2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:37.951568 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.951526 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 11:49:37.953563 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.952712 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 11:49:37.953563 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.952760 2555 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 11:49:37.953563 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.952787 2555 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 11:49:37.953563 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:37.952828 2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 11:49:37.955363 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.955347 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:49:37.977508 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.977481 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:49:37.978435 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.978417 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:49:37.978519 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.978450 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:49:37.978519 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.978463 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:49:37.978519 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.978485 2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-190.ec2.internal" Jul 22 11:49:37.987735 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:37.987718 2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-190.ec2.internal" Jul 22 11:49:37.987787 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:37.987742 2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-190.ec2.internal\": node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:38.003492 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:38.003472 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:38.053068 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.053031 2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-190.ec2.internal"] Jul 22 11:49:38.053164 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.053131 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:49:38.054098 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.054083 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:49:38.054180 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.054127 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:49:38.054180 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.054137 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:49:38.056358 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.056346 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:49:38.056512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.056499 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.056556 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.056535 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:49:38.057019 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.057003 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:49:38.057019 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.057015 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:49:38.057129 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.057033 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:49:38.057129 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.057044 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:49:38.057129 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.057034 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:49:38.057129 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.057128 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:49:38.059259 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.059245 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.059318 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.059269 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:49:38.059955 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.059940 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:49:38.060021 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.059967 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:49:38.060021 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.059979 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:49:38.078930 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:38.078906 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-190.ec2.internal\" not found" node="ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.083451 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:38.083432 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-190.ec2.internal\" not found" node="ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.104391 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:38.104366 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:38.148529 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.148497 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3972d1ac493d357c8ffc3d82a19aef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal\" (UID: \"2c3972d1ac493d357c8ffc3d82a19aef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.148617 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.148533 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0028e82bce45bbfeac23212d15624aa-config\") pod \"kube-apiserver-proxy-ip-10-0-130-190.ec2.internal\" (UID: \"a0028e82bce45bbfeac23212d15624aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.148617 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.148552 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c3972d1ac493d357c8ffc3d82a19aef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal\" (UID: \"2c3972d1ac493d357c8ffc3d82a19aef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.204974 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:38.204906 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:38.249181 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.249138 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c3972d1ac493d357c8ffc3d82a19aef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal\" (UID: \"2c3972d1ac493d357c8ffc3d82a19aef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.249253 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.249197 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3972d1ac493d357c8ffc3d82a19aef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal\" (UID: \"2c3972d1ac493d357c8ffc3d82a19aef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.249253 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.249215 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0028e82bce45bbfeac23212d15624aa-config\") pod \"kube-apiserver-proxy-ip-10-0-130-190.ec2.internal\" (UID: \"a0028e82bce45bbfeac23212d15624aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.249313 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.249251 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0028e82bce45bbfeac23212d15624aa-config\") pod \"kube-apiserver-proxy-ip-10-0-130-190.ec2.internal\" (UID: \"a0028e82bce45bbfeac23212d15624aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.249313 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.249262 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3972d1ac493d357c8ffc3d82a19aef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal\" (UID: \"2c3972d1ac493d357c8ffc3d82a19aef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.249313 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.249262 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c3972d1ac493d357c8ffc3d82a19aef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal\" (UID: \"2c3972d1ac493d357c8ffc3d82a19aef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.305506 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:38.305476 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:38.382834 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.382808 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.386772 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.386753 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.ec2.internal" Jul 22 11:49:38.406003 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:38.405970 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:38.506663 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:38.506577 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:38.523748 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.523725 2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 11:49:38.524311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.523884 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 11:49:38.604653 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.604620 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 11:44:37 +0000 UTC" deadline="2028-03-30 09:58:56.066210846 +0000 UTC" Jul 22 11:49:38.604653 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.604646 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14806h9m17.461567742s" Jul 22 11:49:38.606755 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:38.606725 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:38.707577 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:38.707540 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:38.808260 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:38.808180 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:38.843567 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.843536 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:49:38.909006 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:38.908977 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:38.944783 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:38.944757 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:49:39.010010 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:39.009975 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:39.027053 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:39.027016 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0028e82bce45bbfeac23212d15624aa.slice/crio-7f4f5c1c4df23424a9367c514c6f996020837d4368db19dea381a8e3d5db0ecf WatchSource:0}: Error finding container 7f4f5c1c4df23424a9367c514c6f996020837d4368db19dea381a8e3d5db0ecf: Status 404 returned error can't find the container with id 7f4f5c1c4df23424a9367c514c6f996020837d4368db19dea381a8e3d5db0ecf Jul 22 11:49:39.027355 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:39.027338 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3972d1ac493d357c8ffc3d82a19aef.slice/crio-40f994a76717d3a2ed3565af41a745ae108f6fca63a1612c9dde1e4383a113a2 WatchSource:0}: Error finding container 40f994a76717d3a2ed3565af41a745ae108f6fca63a1612c9dde1e4383a113a2: Status 404 returned error can't find the container with id 40f994a76717d3a2ed3565af41a745ae108f6fca63a1612c9dde1e4383a113a2 Jul 22 11:49:39.033032 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.033018 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 11:49:39.110666 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:39.110586 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-190.ec2.internal\" not found" Jul 22 11:49:39.135166 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.135145 2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:49:39.202511 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.202484 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal" Jul 22 11:49:39.214927 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.214901 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 11:49:39.215883 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.215870 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.ec2.internal" Jul 22 11:49:39.225691 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.225678 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 11:49:39.513257 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.513180 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:49:39.568888 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.568852 2555 apiserver.go:52] "Watching apiserver" Jul 22 11:49:39.577181 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.576266 2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 11:49:39.577181 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.576771 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-4s464","kube-system/kube-apiserver-proxy-ip-10-0-130-190.ec2.internal","openshift-image-registry/node-ca-qrmnr","openshift-multus/multus-additional-cni-plugins-4mfbk","openshift-multus/multus-dc7km","openshift-multus/network-metrics-daemon-vk2qp","openshift-network-diagnostics/network-check-target-78b62","openshift-network-operator/iptables-alerter-z8gks","kube-system/global-pull-secret-syncer-t4dxk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n","openshift-cluster-node-tuning-operator/tuned-mxvfx","openshift-dns/node-resolver-vclwc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-6mthh"] Jul 22 11:49:39.579949 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.579406 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4s464" Jul 22 11:49:39.582522 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.581970 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z8gks" Jul 22 11:49:39.582855 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.582697 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 11:49:39.582855 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.582720 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7x88j\"" Jul 22 11:49:39.582855 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.582751 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 11:49:39.584622 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.584179 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:39.584622 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:39.584244 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:49:39.585484 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.585433 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 11:49:39.585582 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.585509 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 11:49:39.585672 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.585656 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jxqpl\"" Jul 22 11:49:39.585729 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.585703 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:49:39.587159 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.587140 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.590022 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.590003 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.590971 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.590862 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:39.590971 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:39.590935 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:49:39.592063 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.591849 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 11:49:39.592063 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.591933 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 11:49:39.592063 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.591857 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 11:49:39.592449 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.592138 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 11:49:39.592449 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.592164 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 11:49:39.592449 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.592443 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6lml9\"" Jul 22 11:49:39.592601 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.592579 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 11:49:39.593148 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.593043 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrmnr" Jul 22 11:49:39.594215 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.593757 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zxp6h\"" Jul 22 11:49:39.595544 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.595530 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:39.595747 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:39.595707 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:49:39.596936 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.596310 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 11:49:39.596936 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.596568 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 11:49:39.596936 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.596779 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 11:49:39.597504 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.597352 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-czj6w\"" Jul 22 11:49:39.600332 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.599366 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.601940 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.601663 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 11:49:39.601940 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.601905 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 11:49:39.606699 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.604802 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x4x8z\"" Jul 22 11:49:39.606699 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.605157 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 11:49:39.606699 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.605262 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.606699 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.605394 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 11:49:39.608469 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.607455 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:49:39.608469 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.607962 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rc6t5\"" Jul 22 11:49:39.608469 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.607976 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 11:49:39.609723 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.609704 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vclwc" Jul 22 11:49:39.612270 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.612253 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.613069 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.612993 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qm6jt\"" Jul 22 11:49:39.613181 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.613124 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 11:49:39.613265 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.613252 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 11:49:39.614275 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.614258 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 11:49:39.615574 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.615396 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 11:49:39.615747 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.615732 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tkf6l\"" Jul 22 11:49:39.616907 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.616889 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 11:49:39.617191 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.617173 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 11:49:39.617691 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.617189 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 11:49:39.617691 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.617265 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 11:49:39.618064 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.617895 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 11:49:39.634684 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.634610 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fj8r9" Jul 22 11:49:39.644462 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.644085 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fj8r9" Jul 22 11:49:39.657156 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657129 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-sysconfig\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.657261 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657170 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/57933426-ecf3-46ae-87b9-9a9f8329462e-agent-certs\") pod \"konnectivity-agent-4s464\" (UID: \"57933426-ecf3-46ae-87b9-9a9f8329462e\") " pod="kube-system/konnectivity-agent-4s464" Jul 22 11:49:39.657261 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657194 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20f96103-fe3f-4f0f-8f59-351a00064147-host\") pod \"node-ca-qrmnr\" (UID: \"20f96103-fe3f-4f0f-8f59-351a00064147\") " pod="openshift-image-registry/node-ca-qrmnr" Jul 22 11:49:39.657261 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657225 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-etc-selinux\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.657261 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657248 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-kubernetes\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.657408 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657274 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsckn\" (UniqueName: \"kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn\") pod \"network-check-target-78b62\" (UID: \"07c0d7ac-5517-4327-a6cc-4c7c105ac449\") " pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:39.657408 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657309 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff6fe23-8563-41c9-a23b-4e62b98ffa1f-host-slash\") pod \"iptables-alerter-z8gks\" (UID: \"bff6fe23-8563-41c9-a23b-4e62b98ffa1f\") " pod="openshift-network-operator/iptables-alerter-z8gks" Jul 22 11:49:39.657408 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657335 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-system-cni-dir\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.657408 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657358 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-var-lib-kubelet\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.657408 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657383 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs\") pod \"network-metrics-daemon-vk2qp\" (UID: \"5d43201c-5806-4536-8da3-8ad8fa286eac\") " pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:39.657561 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657407 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-sysctl-conf\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.657561 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657429 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cznhk\" (UniqueName: \"kubernetes.io/projected/20f96103-fe3f-4f0f-8f59-351a00064147-kube-api-access-cznhk\") pod \"node-ca-qrmnr\" (UID: \"20f96103-fe3f-4f0f-8f59-351a00064147\") " pod="openshift-image-registry/node-ca-qrmnr" Jul 22 11:49:39.657561 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657472 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-run-k8s-cni-cncf-io\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.657561 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657496 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-sys-fs\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.657561 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657530 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-run\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.657811 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657571 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-host\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.657811 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657595 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bff6fe23-8563-41c9-a23b-4e62b98ffa1f-iptables-alerter-script\") pod \"iptables-alerter-z8gks\" (UID: \"bff6fe23-8563-41c9-a23b-4e62b98ffa1f\") " pod="openshift-network-operator/iptables-alerter-z8gks" Jul 22 11:49:39.657811 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657643 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3cb469cd-363d-4822-906b-5a00b7c0e230-os-release\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.657811 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657665 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-cnibin\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.657811 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657687 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-etc-kubernetes\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.657811 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657730 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-run-netns\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.657811 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657755 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-socket-dir\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.657811 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657777 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65n7x\" (UniqueName: \"kubernetes.io/projected/bff6fe23-8563-41c9-a23b-4e62b98ffa1f-kube-api-access-65n7x\") pod \"iptables-alerter-z8gks\" (UID: \"bff6fe23-8563-41c9-a23b-4e62b98ffa1f\") " pod="openshift-network-operator/iptables-alerter-z8gks" Jul 22 11:49:39.657811 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657802 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cb469cd-363d-4822-906b-5a00b7c0e230-system-cni-dir\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.658315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657834 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsxqs\" (UniqueName: \"kubernetes.io/projected/823c16cc-5e05-42e0-a7e4-bb57adbb2314-kube-api-access-bsxqs\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.658315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657868 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/57933426-ecf3-46ae-87b9-9a9f8329462e-konnectivity-ca\") pod \"konnectivity-agent-4s464\" (UID: \"57933426-ecf3-46ae-87b9-9a9f8329462e\") " pod="kube-system/konnectivity-agent-4s464" Jul 22 11:49:39.658315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657922 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:39.658315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657953 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbnbs\" (UniqueName: \"kubernetes.io/projected/3cb469cd-363d-4822-906b-5a00b7c0e230-kube-api-access-hbnbs\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.658315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.657980 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-run-multus-certs\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.658315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658016 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-modprobe-d\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.658315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658039 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-sysctl-d\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.658315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658062 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/823c16cc-5e05-42e0-a7e4-bb57adbb2314-tmp\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.658315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658094 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3cb469cd-363d-4822-906b-5a00b7c0e230-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.658315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658137 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-var-lib-cni-bin\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.658315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658164 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20f96103-fe3f-4f0f-8f59-351a00064147-serviceca\") pod \"node-ca-qrmnr\" (UID: \"20f96103-fe3f-4f0f-8f59-351a00064147\") " pod="openshift-image-registry/node-ca-qrmnr" Jul 22 11:49:39.658315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658187 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5465cf73-cdb9-46e8-ad04-70550bc615a8-kubelet-config\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:39.658315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658211 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-registration-dir\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658233 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41bf461d-75a8-4bed-8d22-bb7cdd9bf266-tmp-dir\") pod \"node-resolver-vclwc\" (UID: \"41bf461d-75a8-4bed-8d22-bb7cdd9bf266\") " pod="openshift-dns/node-resolver-vclwc" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658270 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-multus-cni-dir\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658294 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-multus-socket-dir-parent\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658318 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-var-lib-cni-multus\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658354 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cb469cd-363d-4822-906b-5a00b7c0e230-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658378 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d49ef242-9548-4f15-a5b6-b3278c34e011-cni-binary-copy\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658403 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzs8t\" (UniqueName: \"kubernetes.io/projected/d49ef242-9548-4f15-a5b6-b3278c34e011-kube-api-access-gzs8t\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658428 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-systemd\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658453 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-os-release\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658476 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-multus-conf-dir\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658500 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d49ef242-9548-4f15-a5b6-b3278c34e011-multus-daemon-config\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658522 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658603 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-sys\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.658903 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658627 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-lib-modules\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.659534 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658652 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsrzh\" (UniqueName: \"kubernetes.io/projected/41bf461d-75a8-4bed-8d22-bb7cdd9bf266-kube-api-access-dsrzh\") pod \"node-resolver-vclwc\" (UID: \"41bf461d-75a8-4bed-8d22-bb7cdd9bf266\") " pod="openshift-dns/node-resolver-vclwc" Jul 22 11:49:39.659534 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658675 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-device-dir\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.659534 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658697 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41bf461d-75a8-4bed-8d22-bb7cdd9bf266-hosts-file\") pod \"node-resolver-vclwc\" (UID: \"41bf461d-75a8-4bed-8d22-bb7cdd9bf266\") " pod="openshift-dns/node-resolver-vclwc" Jul 22 11:49:39.659534 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658718 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cb469cd-363d-4822-906b-5a00b7c0e230-cnibin\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.659534 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658744 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cb469cd-363d-4822-906b-5a00b7c0e230-cni-binary-copy\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.659534 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658771 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3cb469cd-363d-4822-906b-5a00b7c0e230-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.659534 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658796 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wq5h\" (UniqueName: \"kubernetes.io/projected/5d43201c-5806-4536-8da3-8ad8fa286eac-kube-api-access-4wq5h\") pod \"network-metrics-daemon-vk2qp\" (UID: \"5d43201c-5806-4536-8da3-8ad8fa286eac\") " pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:39.659534 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658827 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5465cf73-cdb9-46e8-ad04-70550bc615a8-dbus\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:39.659534 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658871 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-var-lib-kubelet\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.659534 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658902 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-tuned\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.659534 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658931 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-hostroot\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.659534 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.658973 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvpdj\" (UniqueName: \"kubernetes.io/projected/83fbbb08-6d3f-455d-ac19-13386bb78ed5-kube-api-access-dvpdj\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.705620 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.705501 2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 11:49:39.760207 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760171 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-kubelet\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.760381 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760222 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-modprobe-d\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.760381 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760262 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-sysctl-d\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.760381 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760285 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/823c16cc-5e05-42e0-a7e4-bb57adbb2314-tmp\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.760381 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760307 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3cb469cd-363d-4822-906b-5a00b7c0e230-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.760381 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760331 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-var-lib-cni-bin\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.760381 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760373 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20f96103-fe3f-4f0f-8f59-351a00064147-serviceca\") pod \"node-ca-qrmnr\" (UID: \"20f96103-fe3f-4f0f-8f59-351a00064147\") " pod="openshift-image-registry/node-ca-qrmnr" Jul 22 11:49:39.760660 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760387 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-modprobe-d\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.760660 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760416 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/159c1c65-a08a-4f75-8a16-f0469ae1428f-ovnkube-config\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.760660 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760456 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/159c1c65-a08a-4f75-8a16-f0469ae1428f-ovnkube-script-lib\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.760660 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760497 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5465cf73-cdb9-46e8-ad04-70550bc615a8-kubelet-config\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:39.760660 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760548 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-registration-dir\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.760660 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760569 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41bf461d-75a8-4bed-8d22-bb7cdd9bf266-tmp-dir\") pod \"node-resolver-vclwc\" (UID: \"41bf461d-75a8-4bed-8d22-bb7cdd9bf266\") " pod="openshift-dns/node-resolver-vclwc" Jul 22 11:49:39.760660 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760597 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-multus-cni-dir\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.760660 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760620 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-multus-socket-dir-parent\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.760660 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760651 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-var-lib-cni-multus\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.761086 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760674 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cb469cd-363d-4822-906b-5a00b7c0e230-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.761086 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760697 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d49ef242-9548-4f15-a5b6-b3278c34e011-cni-binary-copy\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.761086 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760718 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzs8t\" (UniqueName: \"kubernetes.io/projected/d49ef242-9548-4f15-a5b6-b3278c34e011-kube-api-access-gzs8t\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.761086 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760747 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-cni-bin\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.761086 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760768 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/159c1c65-a08a-4f75-8a16-f0469ae1428f-ovn-node-metrics-cert\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.761086 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760791 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-systemd\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.761086 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.760813 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-os-release\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.761396 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.761344 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-multus-conf-dir\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.761450 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.761414 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d49ef242-9548-4f15-a5b6-b3278c34e011-multus-daemon-config\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.761508 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.761449 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/159c1c65-a08a-4f75-8a16-f0469ae1428f-env-overrides\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.761508 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.761477 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.761605 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.761517 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-sys\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.761605 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.761549 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-lib-modules\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.761605 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.761591 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsrzh\" (UniqueName: \"kubernetes.io/projected/41bf461d-75a8-4bed-8d22-bb7cdd9bf266-kube-api-access-dsrzh\") pod \"node-resolver-vclwc\" (UID: \"41bf461d-75a8-4bed-8d22-bb7cdd9bf266\") " pod="openshift-dns/node-resolver-vclwc" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.761826 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-run-systemd\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.761859 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-device-dir\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.761918 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41bf461d-75a8-4bed-8d22-bb7cdd9bf266-hosts-file\") pod \"node-resolver-vclwc\" (UID: \"41bf461d-75a8-4bed-8d22-bb7cdd9bf266\") " pod="openshift-dns/node-resolver-vclwc" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.761947 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cb469cd-363d-4822-906b-5a00b7c0e230-cnibin\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.761978 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cb469cd-363d-4822-906b-5a00b7c0e230-cni-binary-copy\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.762009 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3cb469cd-363d-4822-906b-5a00b7c0e230-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.762070 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wq5h\" (UniqueName: \"kubernetes.io/projected/5d43201c-5806-4536-8da3-8ad8fa286eac-kube-api-access-4wq5h\") pod \"network-metrics-daemon-vk2qp\" (UID: \"5d43201c-5806-4536-8da3-8ad8fa286eac\") " pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.762138 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-node-log\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.762176 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5465cf73-cdb9-46e8-ad04-70550bc615a8-dbus\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.762205 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-slash\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.762244 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-log-socket\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.762281 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-var-lib-kubelet\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.762304 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-tuned\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.765512 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.762339 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-hostroot\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.766216 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.762381 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-cni-netd\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.766216 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.762429 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvpdj\" (UniqueName: \"kubernetes.io/projected/83fbbb08-6d3f-455d-ac19-13386bb78ed5-kube-api-access-dvpdj\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.766216 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.762536 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-sysconfig\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.766216 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.762581 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/57933426-ecf3-46ae-87b9-9a9f8329462e-agent-certs\") pod \"konnectivity-agent-4s464\" (UID: \"57933426-ecf3-46ae-87b9-9a9f8329462e\") " pod="kube-system/konnectivity-agent-4s464" Jul 22 11:49:39.766216 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.763132 2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 11:49:39.766216 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.763516 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-sysctl-d\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.766216 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.763580 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-multus-cni-dir\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.766216 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.764332 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3cb469cd-363d-4822-906b-5a00b7c0e230-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.766216 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.764948 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20f96103-fe3f-4f0f-8f59-351a00064147-host\") pod \"node-ca-qrmnr\" (UID: \"20f96103-fe3f-4f0f-8f59-351a00064147\") " pod="openshift-image-registry/node-ca-qrmnr" Jul 22 11:49:39.767011 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.766986 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-lib-modules\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.767137 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.767059 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41bf461d-75a8-4bed-8d22-bb7cdd9bf266-tmp-dir\") pod \"node-resolver-vclwc\" (UID: \"41bf461d-75a8-4bed-8d22-bb7cdd9bf266\") " pod="openshift-dns/node-resolver-vclwc" Jul 22 11:49:39.767137 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.767096 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-multus-socket-dir-parent\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.767253 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.767173 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-var-lib-cni-multus\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.767348 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.767326 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cb469cd-363d-4822-906b-5a00b7c0e230-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.769210 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.767831 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d49ef242-9548-4f15-a5b6-b3278c34e011-cni-binary-copy\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.769210 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.768400 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-os-release\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.769210 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.768535 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-multus-conf-dir\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.769210 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.768679 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-systemd\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.769210 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.768722 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.769210 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.768800 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-sys\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.769210 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.769072 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-device-dir\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.769210 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.769148 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41bf461d-75a8-4bed-8d22-bb7cdd9bf266-hosts-file\") pod \"node-resolver-vclwc\" (UID: \"41bf461d-75a8-4bed-8d22-bb7cdd9bf266\") " pod="openshift-dns/node-resolver-vclwc" Jul 22 11:49:39.769210 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.769198 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cb469cd-363d-4822-906b-5a00b7c0e230-cnibin\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.769686 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.769215 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d49ef242-9548-4f15-a5b6-b3278c34e011-multus-daemon-config\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.769789 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.769764 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cb469cd-363d-4822-906b-5a00b7c0e230-cni-binary-copy\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.771128 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.770298 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20f96103-fe3f-4f0f-8f59-351a00064147-host\") pod \"node-ca-qrmnr\" (UID: \"20f96103-fe3f-4f0f-8f59-351a00064147\") " pod="openshift-image-registry/node-ca-qrmnr" Jul 22 11:49:39.771128 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.770453 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3cb469cd-363d-4822-906b-5a00b7c0e230-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.771128 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.770540 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-var-lib-kubelet\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.771128 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.770545 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5465cf73-cdb9-46e8-ad04-70550bc615a8-dbus\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:39.771128 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.770597 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-hostroot\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.771128 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.770777 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/57933426-ecf3-46ae-87b9-9a9f8329462e-agent-certs\") pod \"konnectivity-agent-4s464\" (UID: \"57933426-ecf3-46ae-87b9-9a9f8329462e\") " pod="kube-system/konnectivity-agent-4s464" Jul 22 11:49:39.771128 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.770808 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-sysconfig\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.771128 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.770898 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-var-lib-cni-bin\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.771128 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.770961 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5465cf73-cdb9-46e8-ad04-70550bc615a8-kubelet-config\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:39.771128 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771013 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-run-netns\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.771128 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771069 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-run-ovn-kubernetes\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.771128 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771141 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-etc-selinux\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.771733 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771189 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-kubernetes\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.771733 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771198 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-registration-dir\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.771733 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771216 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsckn\" (UniqueName: \"kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn\") pod \"network-check-target-78b62\" (UID: \"07c0d7ac-5517-4327-a6cc-4c7c105ac449\") " pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:39.771733 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771297 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-kubernetes\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.771733 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771378 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-etc-selinux\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.771733 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771408 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20f96103-fe3f-4f0f-8f59-351a00064147-serviceca\") pod \"node-ca-qrmnr\" (UID: \"20f96103-fe3f-4f0f-8f59-351a00064147\") " pod="openshift-image-registry/node-ca-qrmnr" Jul 22 11:49:39.771733 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771425 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff6fe23-8563-41c9-a23b-4e62b98ffa1f-host-slash\") pod \"iptables-alerter-z8gks\" (UID: \"bff6fe23-8563-41c9-a23b-4e62b98ffa1f\") " pod="openshift-network-operator/iptables-alerter-z8gks" Jul 22 11:49:39.771733 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771475 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-system-cni-dir\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.771733 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771547 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-system-cni-dir\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.771733 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771629 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff6fe23-8563-41c9-a23b-4e62b98ffa1f-host-slash\") pod \"iptables-alerter-z8gks\" (UID: \"bff6fe23-8563-41c9-a23b-4e62b98ffa1f\") " pod="openshift-network-operator/iptables-alerter-z8gks" Jul 22 11:49:39.771733 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771676 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-var-lib-kubelet\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.771733 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771719 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs\") pod \"network-metrics-daemon-vk2qp\" (UID: \"5d43201c-5806-4536-8da3-8ad8fa286eac\") " pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:39.772388 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.771765 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-var-lib-kubelet\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.772388 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:39.771867 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:49:39.772487 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:39.772410 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs podName:5d43201c-5806-4536-8da3-8ad8fa286eac nodeName:}" failed. No retries permitted until 2026-07-22 11:49:40.272387024 +0000 UTC m=+3.021598626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs") pod "network-metrics-daemon-vk2qp" (UID: "5d43201c-5806-4536-8da3-8ad8fa286eac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:49:39.772680 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.772650 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-sysctl-conf\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.772680 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.772250 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-sysctl-conf\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.772802 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.772724 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cznhk\" (UniqueName: \"kubernetes.io/projected/20f96103-fe3f-4f0f-8f59-351a00064147-kube-api-access-cznhk\") pod \"node-ca-qrmnr\" (UID: \"20f96103-fe3f-4f0f-8f59-351a00064147\") " pod="openshift-image-registry/node-ca-qrmnr" Jul 22 11:49:39.772971 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.772948 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/823c16cc-5e05-42e0-a7e4-bb57adbb2314-etc-tuned\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773142 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-var-lib-openvswitch\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773294 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-run-ovn\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773321 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7xxw\" (UniqueName: \"kubernetes.io/projected/159c1c65-a08a-4f75-8a16-f0469ae1428f-kube-api-access-z7xxw\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773349 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-run-k8s-cni-cncf-io\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773392 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-sys-fs\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773414 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-run\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773445 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-run-k8s-cni-cncf-io\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773461 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-host\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773486 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bff6fe23-8563-41c9-a23b-4e62b98ffa1f-iptables-alerter-script\") pod \"iptables-alerter-z8gks\" (UID: \"bff6fe23-8563-41c9-a23b-4e62b98ffa1f\") " pod="openshift-network-operator/iptables-alerter-z8gks" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773509 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-sys-fs\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773511 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3cb469cd-363d-4822-906b-5a00b7c0e230-os-release\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773555 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-host\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773596 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/823c16cc-5e05-42e0-a7e4-bb57adbb2314-run\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773548 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-cnibin\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.774626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773632 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-etc-kubernetes\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773656 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-run-netns\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773693 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-socket-dir\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773721 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-etc-openvswitch\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773846 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-etc-kubernetes\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773914 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3cb469cd-363d-4822-906b-5a00b7c0e230-os-release\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.773962 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-cnibin\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.774194 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bff6fe23-8563-41c9-a23b-4e62b98ffa1f-iptables-alerter-script\") pod \"iptables-alerter-z8gks\" (UID: \"bff6fe23-8563-41c9-a23b-4e62b98ffa1f\") " pod="openshift-network-operator/iptables-alerter-z8gks" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.774297 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/83fbbb08-6d3f-455d-ac19-13386bb78ed5-socket-dir\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.774338 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-run-netns\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.774511 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-run-openvswitch\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.774555 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.775086 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65n7x\" (UniqueName: \"kubernetes.io/projected/bff6fe23-8563-41c9-a23b-4e62b98ffa1f-kube-api-access-65n7x\") pod \"iptables-alerter-z8gks\" (UID: \"bff6fe23-8563-41c9-a23b-4e62b98ffa1f\") " pod="openshift-network-operator/iptables-alerter-z8gks" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.775243 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cb469cd-363d-4822-906b-5a00b7c0e230-system-cni-dir\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.777776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.775458 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/823c16cc-5e05-42e0-a7e4-bb57adbb2314-tmp\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.778734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.775521 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cb469cd-363d-4822-906b-5a00b7c0e230-system-cni-dir\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.778734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.775804 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-systemd-units\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.778734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.775955 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsxqs\" (UniqueName: \"kubernetes.io/projected/823c16cc-5e05-42e0-a7e4-bb57adbb2314-kube-api-access-bsxqs\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.778734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.776101 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/57933426-ecf3-46ae-87b9-9a9f8329462e-konnectivity-ca\") pod \"konnectivity-agent-4s464\" (UID: \"57933426-ecf3-46ae-87b9-9a9f8329462e\") " pod="kube-system/konnectivity-agent-4s464" Jul 22 11:49:39.778734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.776159 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:39.778734 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:39.776487 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:49:39.778734 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:39.776598 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret podName:5465cf73-cdb9-46e8-ad04-70550bc615a8 nodeName:}" failed. No retries permitted until 2026-07-22 11:49:40.27658059 +0000 UTC m=+3.025792175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret") pod "global-pull-secret-syncer-t4dxk" (UID: "5465cf73-cdb9-46e8-ad04-70550bc615a8") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:49:39.778734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.776855 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/57933426-ecf3-46ae-87b9-9a9f8329462e-konnectivity-ca\") pod \"konnectivity-agent-4s464\" (UID: \"57933426-ecf3-46ae-87b9-9a9f8329462e\") " pod="kube-system/konnectivity-agent-4s464" Jul 22 11:49:39.778734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.776183 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnbs\" (UniqueName: \"kubernetes.io/projected/3cb469cd-363d-4822-906b-5a00b7c0e230-kube-api-access-hbnbs\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.778734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.776932 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-run-multus-certs\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.778734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.777013 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d49ef242-9548-4f15-a5b6-b3278c34e011-host-run-multus-certs\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.778734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.777940 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsrzh\" (UniqueName: \"kubernetes.io/projected/41bf461d-75a8-4bed-8d22-bb7cdd9bf266-kube-api-access-dsrzh\") pod \"node-resolver-vclwc\" (UID: \"41bf461d-75a8-4bed-8d22-bb7cdd9bf266\") " pod="openshift-dns/node-resolver-vclwc" Jul 22 11:49:39.778734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.778350 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzs8t\" (UniqueName: \"kubernetes.io/projected/d49ef242-9548-4f15-a5b6-b3278c34e011-kube-api-access-gzs8t\") pod \"multus-dc7km\" (UID: \"d49ef242-9548-4f15-a5b6-b3278c34e011\") " pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.781061 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:39.781041 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:49:39.781166 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:39.781070 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:49:39.781166 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:39.781084 2555 projected.go:196] Error preparing data for projected volume kube-api-access-zsckn for pod openshift-network-diagnostics/network-check-target-78b62: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:49:39.781387 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:39.781244 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn podName:07c0d7ac-5517-4327-a6cc-4c7c105ac449 nodeName:}" failed. No retries permitted until 2026-07-22 11:49:40.281228807 +0000 UTC m=+3.030440392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zsckn" (UniqueName: "kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn") pod "network-check-target-78b62" (UID: "07c0d7ac-5517-4327-a6cc-4c7c105ac449") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:49:39.781978 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.781943 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvpdj\" (UniqueName: \"kubernetes.io/projected/83fbbb08-6d3f-455d-ac19-13386bb78ed5-kube-api-access-dvpdj\") pod \"aws-ebs-csi-driver-node-lws4n\" (UID: \"83fbbb08-6d3f-455d-ac19-13386bb78ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.783382 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.783339 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wq5h\" (UniqueName: \"kubernetes.io/projected/5d43201c-5806-4536-8da3-8ad8fa286eac-kube-api-access-4wq5h\") pod \"network-metrics-daemon-vk2qp\" (UID: \"5d43201c-5806-4536-8da3-8ad8fa286eac\") " pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:39.783653 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.783605 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65n7x\" (UniqueName: \"kubernetes.io/projected/bff6fe23-8563-41c9-a23b-4e62b98ffa1f-kube-api-access-65n7x\") pod \"iptables-alerter-z8gks\" (UID: \"bff6fe23-8563-41c9-a23b-4e62b98ffa1f\") " pod="openshift-network-operator/iptables-alerter-z8gks" Jul 22 11:49:39.785496 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.785082 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cznhk\" (UniqueName: \"kubernetes.io/projected/20f96103-fe3f-4f0f-8f59-351a00064147-kube-api-access-cznhk\") pod \"node-ca-qrmnr\" (UID: \"20f96103-fe3f-4f0f-8f59-351a00064147\") " pod="openshift-image-registry/node-ca-qrmnr" Jul 22 11:49:39.785496 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.785471 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsxqs\" (UniqueName: \"kubernetes.io/projected/823c16cc-5e05-42e0-a7e4-bb57adbb2314-kube-api-access-bsxqs\") pod \"tuned-mxvfx\" (UID: \"823c16cc-5e05-42e0-a7e4-bb57adbb2314\") " pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.786000 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.785978 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnbs\" (UniqueName: \"kubernetes.io/projected/3cb469cd-363d-4822-906b-5a00b7c0e230-kube-api-access-hbnbs\") pod \"multus-additional-cni-plugins-4mfbk\" (UID: \"3cb469cd-363d-4822-906b-5a00b7c0e230\") " pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.878207 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878174 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-etc-openvswitch\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878207 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878215 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-run-openvswitch\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878441 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878240 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878441 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878265 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-systemd-units\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878441 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878303 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-kubelet\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878441 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878302 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-etc-openvswitch\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878441 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878332 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/159c1c65-a08a-4f75-8a16-f0469ae1428f-ovnkube-config\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878441 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878361 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/159c1c65-a08a-4f75-8a16-f0469ae1428f-ovnkube-script-lib\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878441 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878374 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878441 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878390 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-cni-bin\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878441 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878414 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/159c1c65-a08a-4f75-8a16-f0469ae1428f-ovn-node-metrics-cert\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878441 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878418 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-run-openvswitch\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878441 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878442 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/159c1c65-a08a-4f75-8a16-f0469ae1428f-env-overrides\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878982 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878472 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-run-systemd\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878982 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878505 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-node-log\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878982 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878530 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-slash\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878982 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878553 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-log-socket\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878982 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878595 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-cni-netd\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878982 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878623 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-run-netns\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878982 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878648 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-run-ovn-kubernetes\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878982 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878702 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-var-lib-openvswitch\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878982 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878726 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-run-ovn\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.878982 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.878751 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7xxw\" (UniqueName: \"kubernetes.io/projected/159c1c65-a08a-4f75-8a16-f0469ae1428f-kube-api-access-z7xxw\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879030 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-slash\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879120 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-cni-netd\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879188 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-run-netns\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879231 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-run-ovn-kubernetes\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879272 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-var-lib-openvswitch\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879309 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-run-ovn\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879370 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-cni-bin\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879407 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-systemd-units\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879469 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/159c1c65-a08a-4f75-8a16-f0469ae1428f-env-overrides\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879516 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-run-systemd\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879990 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879542 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-host-kubelet\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879990 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879565 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-node-log\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879990 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879587 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/159c1c65-a08a-4f75-8a16-f0469ae1428f-log-socket\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879990 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879636 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/159c1c65-a08a-4f75-8a16-f0469ae1428f-ovnkube-script-lib\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.879990 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.879785 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/159c1c65-a08a-4f75-8a16-f0469ae1428f-ovnkube-config\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.882053 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.882030 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/159c1c65-a08a-4f75-8a16-f0469ae1428f-ovn-node-metrics-cert\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.889153 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.889128 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7xxw\" (UniqueName: \"kubernetes.io/projected/159c1c65-a08a-4f75-8a16-f0469ae1428f-kube-api-access-z7xxw\") pod \"ovnkube-node-6mthh\" (UID: \"159c1c65-a08a-4f75-8a16-f0469ae1428f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:39.894957 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.894926 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4s464" Jul 22 11:49:39.905777 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.905744 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z8gks" Jul 22 11:49:39.919196 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.919163 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4mfbk" Jul 22 11:49:39.924181 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.924151 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dc7km" Jul 22 11:49:39.933612 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.933583 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrmnr" Jul 22 11:49:39.944481 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.944455 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" Jul 22 11:49:39.951437 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.951414 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" Jul 22 11:49:39.957752 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.957675 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal" event={"ID":"2c3972d1ac493d357c8ffc3d82a19aef","Type":"ContainerStarted","Data":"40f994a76717d3a2ed3565af41a745ae108f6fca63a1612c9dde1e4383a113a2"} Jul 22 11:49:39.958916 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.958888 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.ec2.internal" event={"ID":"a0028e82bce45bbfeac23212d15624aa","Type":"ContainerStarted","Data":"7f4f5c1c4df23424a9367c514c6f996020837d4368db19dea381a8e3d5db0ecf"} Jul 22 11:49:39.960004 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.959985 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vclwc" Jul 22 11:49:39.967671 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:39.967650 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:49:40.282051 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.282010 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsckn\" (UniqueName: \"kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn\") pod \"network-check-target-78b62\" (UID: \"07c0d7ac-5517-4327-a6cc-4c7c105ac449\") " pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:40.282051 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.282056 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs\") pod \"network-metrics-daemon-vk2qp\" (UID: \"5d43201c-5806-4536-8da3-8ad8fa286eac\") " pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:40.282300 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.282084 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:40.282300 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:40.282199 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:49:40.282300 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:40.282222 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:49:40.282300 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:40.282252 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:49:40.282300 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:40.282267 2555 projected.go:196] Error preparing data for projected volume kube-api-access-zsckn for pod openshift-network-diagnostics/network-check-target-78b62: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:49:40.282300 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:40.282200 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:49:40.282300 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:40.282255 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret podName:5465cf73-cdb9-46e8-ad04-70550bc615a8 nodeName:}" failed. No retries permitted until 2026-07-22 11:49:41.282242623 +0000 UTC m=+4.031454202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret") pod "global-pull-secret-syncer-t4dxk" (UID: "5465cf73-cdb9-46e8-ad04-70550bc615a8") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:49:40.282641 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:40.282333 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn podName:07c0d7ac-5517-4327-a6cc-4c7c105ac449 nodeName:}" failed. No retries permitted until 2026-07-22 11:49:41.282315181 +0000 UTC m=+4.031526769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zsckn" (UniqueName: "kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn") pod "network-check-target-78b62" (UID: "07c0d7ac-5517-4327-a6cc-4c7c105ac449") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:49:40.282641 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:40.282357 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs podName:5d43201c-5806-4536-8da3-8ad8fa286eac nodeName:}" failed. No retries permitted until 2026-07-22 11:49:41.28234693 +0000 UTC m=+4.031558523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs") pod "network-metrics-daemon-vk2qp" (UID: "5d43201c-5806-4536-8da3-8ad8fa286eac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:49:40.648097 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.648012 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 11:44:39 +0000 UTC" deadline="2028-03-29 16:15:55.376798446 +0000 UTC" Jul 22 11:49:40.648097 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.648043 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14788h26m14.728759396s" Jul 22 11:49:40.856605 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:40.856578 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff6fe23_8563_41c9_a23b_4e62b98ffa1f.slice/crio-d28bd386dc29e24f3ea69655624494510cd7298360cefeea3500f6bd9365744f WatchSource:0}: Error finding container d28bd386dc29e24f3ea69655624494510cd7298360cefeea3500f6bd9365744f: Status 404 returned error can't find the container with id d28bd386dc29e24f3ea69655624494510cd7298360cefeea3500f6bd9365744f Jul 22 11:49:40.876635 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:40.876601 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57933426_ecf3_46ae_87b9_9a9f8329462e.slice/crio-a8bce1ee77284eb5db34d1a0c36126ae8aa35cd718944e17debf5987fe787691 WatchSource:0}: Error finding container a8bce1ee77284eb5db34d1a0c36126ae8aa35cd718944e17debf5987fe787691: Status 404 returned error can't find the container with id a8bce1ee77284eb5db34d1a0c36126ae8aa35cd718944e17debf5987fe787691 Jul 22 11:49:40.879358 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:40.879335 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823c16cc_5e05_42e0_a7e4_bb57adbb2314.slice/crio-8b9efed6c2a7b243e80f66d66de9af4e71f7dc2dcc409f26d96c0d32b9f3e1c4 WatchSource:0}: Error finding container 8b9efed6c2a7b243e80f66d66de9af4e71f7dc2dcc409f26d96c0d32b9f3e1c4: Status 404 returned error can't find the container with id 8b9efed6c2a7b243e80f66d66de9af4e71f7dc2dcc409f26d96c0d32b9f3e1c4 Jul 22 11:49:40.880216 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:40.880193 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb469cd_363d_4822_906b_5a00b7c0e230.slice/crio-556bcdde9324e481714ee93d8067efd523863d9a2897e73ac649e060841a3294 WatchSource:0}: Error finding container 556bcdde9324e481714ee93d8067efd523863d9a2897e73ac649e060841a3294: Status 404 returned error can't find the container with id 556bcdde9324e481714ee93d8067efd523863d9a2897e73ac649e060841a3294 Jul 22 11:49:40.881073 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:40.881050 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49ef242_9548_4f15_a5b6_b3278c34e011.slice/crio-2e9deb401855106bdb7fcabccd2eb26dad00dcce263fc575bbb92b40a2ae0609 WatchSource:0}: Error finding container 2e9deb401855106bdb7fcabccd2eb26dad00dcce263fc575bbb92b40a2ae0609: Status 404 returned error can't find the container with id 2e9deb401855106bdb7fcabccd2eb26dad00dcce263fc575bbb92b40a2ae0609 Jul 22 11:49:40.881967 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:49:40.881892 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159c1c65_a08a_4f75_8a16_f0469ae1428f.slice/crio-cd5e449c682b8d50981b42f2ff9bdb666ea0ecf63f2cf058fc3c921a95a45d08 WatchSource:0}: Error finding container cd5e449c682b8d50981b42f2ff9bdb666ea0ecf63f2cf058fc3c921a95a45d08: Status 404 returned error can't find the container with id cd5e449c682b8d50981b42f2ff9bdb666ea0ecf63f2cf058fc3c921a95a45d08 Jul 22 11:49:40.953662 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.953440 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:40.953807 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:40.953775 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:49:40.961787 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.961762 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vclwc" event={"ID":"41bf461d-75a8-4bed-8d22-bb7cdd9bf266","Type":"ContainerStarted","Data":"3953073fcba1ceb20a0b27f8adedf39f3c464dde18abc741412244cb9f83d948"} Jul 22 11:49:40.963059 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.963030 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" event={"ID":"159c1c65-a08a-4f75-8a16-f0469ae1428f","Type":"ContainerStarted","Data":"cd5e449c682b8d50981b42f2ff9bdb666ea0ecf63f2cf058fc3c921a95a45d08"} Jul 22 11:49:40.964066 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.964047 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dc7km" event={"ID":"d49ef242-9548-4f15-a5b6-b3278c34e011","Type":"ContainerStarted","Data":"2e9deb401855106bdb7fcabccd2eb26dad00dcce263fc575bbb92b40a2ae0609"} Jul 22 11:49:40.965182 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.965158 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4s464" event={"ID":"57933426-ecf3-46ae-87b9-9a9f8329462e","Type":"ContainerStarted","Data":"a8bce1ee77284eb5db34d1a0c36126ae8aa35cd718944e17debf5987fe787691"} Jul 22 11:49:40.966967 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.966944 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z8gks" event={"ID":"bff6fe23-8563-41c9-a23b-4e62b98ffa1f","Type":"ContainerStarted","Data":"d28bd386dc29e24f3ea69655624494510cd7298360cefeea3500f6bd9365744f"} Jul 22 11:49:40.968341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.968301 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mfbk" event={"ID":"3cb469cd-363d-4822-906b-5a00b7c0e230","Type":"ContainerStarted","Data":"556bcdde9324e481714ee93d8067efd523863d9a2897e73ac649e060841a3294"} Jul 22 11:49:40.969470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.969449 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrmnr" event={"ID":"20f96103-fe3f-4f0f-8f59-351a00064147","Type":"ContainerStarted","Data":"58798a6af8879a1a015967f1dd6f2c9a9a622250f4ae94a59d934b8cd6e9c60e"} Jul 22 11:49:40.970646 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.970621 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" event={"ID":"83fbbb08-6d3f-455d-ac19-13386bb78ed5","Type":"ContainerStarted","Data":"4dc32717821ed20c230389bb8667e41264f825ad6deec7a7394a5fb970e007c4"} Jul 22 11:49:40.971849 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:40.971833 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" event={"ID":"823c16cc-5e05-42e0-a7e4-bb57adbb2314","Type":"ContainerStarted","Data":"8b9efed6c2a7b243e80f66d66de9af4e71f7dc2dcc409f26d96c0d32b9f3e1c4"} Jul 22 11:49:41.288847 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:41.287814 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsckn\" (UniqueName: \"kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn\") pod \"network-check-target-78b62\" (UID: \"07c0d7ac-5517-4327-a6cc-4c7c105ac449\") " pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:41.288847 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:41.287866 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs\") pod \"network-metrics-daemon-vk2qp\" (UID: \"5d43201c-5806-4536-8da3-8ad8fa286eac\") " pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:41.288847 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:41.287911 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:41.288847 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:41.288069 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:49:41.288847 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:41.288151 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret podName:5465cf73-cdb9-46e8-ad04-70550bc615a8 nodeName:}" failed. No retries permitted until 2026-07-22 11:49:43.288132408 +0000 UTC m=+6.037343990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret") pod "global-pull-secret-syncer-t4dxk" (UID: "5465cf73-cdb9-46e8-ad04-70550bc615a8") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:49:41.288847 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:41.288588 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:49:41.288847 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:41.288616 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:49:41.288847 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:41.288628 2555 projected.go:196] Error preparing data for projected volume kube-api-access-zsckn for pod openshift-network-diagnostics/network-check-target-78b62: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:49:41.288847 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:41.288672 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn podName:07c0d7ac-5517-4327-a6cc-4c7c105ac449 nodeName:}" failed. No retries permitted until 2026-07-22 11:49:43.288658111 +0000 UTC m=+6.037869693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zsckn" (UniqueName: "kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn") pod "network-check-target-78b62" (UID: "07c0d7ac-5517-4327-a6cc-4c7c105ac449") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:49:41.288847 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:41.288733 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:49:41.288847 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:41.288766 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs podName:5d43201c-5806-4536-8da3-8ad8fa286eac nodeName:}" failed. No retries permitted until 2026-07-22 11:49:43.288753779 +0000 UTC m=+6.037965362 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs") pod "network-metrics-daemon-vk2qp" (UID: "5d43201c-5806-4536-8da3-8ad8fa286eac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:49:41.648937 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:41.648871 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 11:44:39 +0000 UTC" deadline="2028-04-27 13:29:35.436319948 +0000 UTC" Jul 22 11:49:41.648937 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:41.648934 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15481h39m53.787389s" Jul 22 11:49:41.953944 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:41.953280 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:41.953944 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:41.953325 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:41.953944 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:41.953470 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:49:41.953944 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:41.953899 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:49:41.992765 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:41.992182 2555 generic.go:356] "Generic (PLEG): container finished" podID="2c3972d1ac493d357c8ffc3d82a19aef" containerID="59b211fa83666ca7cd45b1cc42851905caa84163a73a1ef49b5840a9787518bf" exitCode=0 Jul 22 11:49:41.992765 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:41.992283 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal" event={"ID":"2c3972d1ac493d357c8ffc3d82a19aef","Type":"ContainerDied","Data":"59b211fa83666ca7cd45b1cc42851905caa84163a73a1ef49b5840a9787518bf"} Jul 22 11:49:42.001386 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:42.001325 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.ec2.internal" event={"ID":"a0028e82bce45bbfeac23212d15624aa","Type":"ContainerStarted","Data":"1cd9b0c35c5173db1d4291fd72897f77163f5e7dab8769192ce7cfcca92baf0d"} Jul 22 11:49:42.953939 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:42.953897 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:42.954507 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:42.954026 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:49:43.039018 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:43.038483 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal" event={"ID":"2c3972d1ac493d357c8ffc3d82a19aef","Type":"ContainerStarted","Data":"b6db1757dc4f117d7757609ce6022a75257adb68c0166d17570b31c58feb235b"} Jul 22 11:49:43.059038 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:43.058712 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.ec2.internal" podStartSLOduration=4.058689061 podStartE2EDuration="4.058689061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:49:42.037064396 +0000 UTC m=+4.786276001" watchObservedRunningTime="2026-07-22 11:49:43.058689061 +0000 UTC m=+5.807900664" Jul 22 11:49:43.307553 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:43.306564 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:43.307553 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:43.306637 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsckn\" (UniqueName: \"kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn\") pod \"network-check-target-78b62\" (UID: \"07c0d7ac-5517-4327-a6cc-4c7c105ac449\") " pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:43.307553 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:43.306664 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs\") pod \"network-metrics-daemon-vk2qp\" (UID: \"5d43201c-5806-4536-8da3-8ad8fa286eac\") " pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:43.307553 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:43.306780 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:49:43.307553 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:43.306841 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs podName:5d43201c-5806-4536-8da3-8ad8fa286eac nodeName:}" failed. No retries permitted until 2026-07-22 11:49:47.306823835 +0000 UTC m=+10.056035423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs") pod "network-metrics-daemon-vk2qp" (UID: "5d43201c-5806-4536-8da3-8ad8fa286eac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:49:43.307553 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:43.307230 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:49:43.307553 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:43.307277 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret podName:5465cf73-cdb9-46e8-ad04-70550bc615a8 nodeName:}" failed. No retries permitted until 2026-07-22 11:49:47.30726166 +0000 UTC m=+10.056473242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret") pod "global-pull-secret-syncer-t4dxk" (UID: "5465cf73-cdb9-46e8-ad04-70550bc615a8") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:49:43.307553 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:43.307355 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:49:43.307553 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:43.307371 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:49:43.307553 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:43.307383 2555 projected.go:196] Error preparing data for projected volume kube-api-access-zsckn for pod openshift-network-diagnostics/network-check-target-78b62: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:49:43.307553 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:43.307417 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn podName:07c0d7ac-5517-4327-a6cc-4c7c105ac449 nodeName:}" failed. No retries permitted until 2026-07-22 11:49:47.307406647 +0000 UTC m=+10.056618230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zsckn" (UniqueName: "kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn") pod "network-check-target-78b62" (UID: "07c0d7ac-5517-4327-a6cc-4c7c105ac449") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:49:43.954144 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:43.953623 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:43.954144 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:43.953768 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:49:43.954674 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:43.954177 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:43.954674 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:43.954296 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:49:44.953352 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:44.953314 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:44.953543 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:44.953468 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:49:45.953486 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:45.953388 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:45.953934 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:45.953513 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:49:45.953934 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:45.953391 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:45.954060 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:45.953977 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:49:46.953620 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:46.953583 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:46.954072 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:46.953702 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:49:47.338750 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:47.338527 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:47.338750 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:47.338602 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsckn\" (UniqueName: \"kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn\") pod \"network-check-target-78b62\" (UID: \"07c0d7ac-5517-4327-a6cc-4c7c105ac449\") " pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:47.338750 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:47.338620 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs\") pod \"network-metrics-daemon-vk2qp\" (UID: \"5d43201c-5806-4536-8da3-8ad8fa286eac\") " pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:47.338750 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:47.338746 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:49:47.339067 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:47.338800 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs podName:5d43201c-5806-4536-8da3-8ad8fa286eac nodeName:}" failed. No retries permitted until 2026-07-22 11:49:55.338783323 +0000 UTC m=+18.087994904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs") pod "network-metrics-daemon-vk2qp" (UID: "5d43201c-5806-4536-8da3-8ad8fa286eac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:49:47.339154 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:47.339133 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:49:47.339198 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:47.339183 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret podName:5465cf73-cdb9-46e8-ad04-70550bc615a8 nodeName:}" failed. No retries permitted until 2026-07-22 11:49:55.339168459 +0000 UTC m=+18.088380043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret") pod "global-pull-secret-syncer-t4dxk" (UID: "5465cf73-cdb9-46e8-ad04-70550bc615a8") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:49:47.339281 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:47.339259 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:49:47.339281 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:47.339278 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:49:47.339470 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:47.339290 2555 projected.go:196] Error preparing data for projected volume kube-api-access-zsckn for pod openshift-network-diagnostics/network-check-target-78b62: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:49:47.339470 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:47.339329 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn podName:07c0d7ac-5517-4327-a6cc-4c7c105ac449 nodeName:}" failed. No retries permitted until 2026-07-22 11:49:55.339318365 +0000 UTC m=+18.088529962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zsckn" (UniqueName: "kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn") pod "network-check-target-78b62" (UID: "07c0d7ac-5517-4327-a6cc-4c7c105ac449") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:49:47.954242 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:47.954207 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:47.954700 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:47.954348 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:49:47.954766 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:47.954742 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:47.954865 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:47.954839 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:49:48.954344 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:48.953822 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:48.954344 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:48.953962 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:49:49.953646 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:49.953615 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:49.953839 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:49.953750 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:49:49.954132 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:49.954076 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:49.954249 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:49.954203 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:49:50.953829 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:50.953789 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:50.954243 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:50.953929 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:49:51.953475 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:51.953438 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:51.953475 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:51.953481 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:51.953694 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:51.953595 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:49:51.953783 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:51.953764 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:49:52.953253 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:52.953213 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:52.953766 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:52.953355 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:49:53.954020 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:53.953977 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:53.954443 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:53.954135 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:49:53.954443 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:53.954169 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:53.954443 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:53.954294 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:49:54.953669 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:54.953634 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:54.953895 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:54.953755 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:49:55.407485 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:55.407450 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:55.407941 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:55.407503 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsckn\" (UniqueName: \"kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn\") pod \"network-check-target-78b62\" (UID: \"07c0d7ac-5517-4327-a6cc-4c7c105ac449\") " pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:55.407941 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:55.407521 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs\") pod \"network-metrics-daemon-vk2qp\" (UID: \"5d43201c-5806-4536-8da3-8ad8fa286eac\") " pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:55.407941 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:55.407570 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:49:55.407941 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:55.407596 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:49:55.407941 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:55.407617 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:49:55.407941 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:55.407638 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:49:55.407941 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:55.407650 2555 projected.go:196] Error preparing data for projected volume kube-api-access-zsckn for pod openshift-network-diagnostics/network-check-target-78b62: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:49:55.407941 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:55.407652 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret podName:5465cf73-cdb9-46e8-ad04-70550bc615a8 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:11.407629426 +0000 UTC m=+34.156841017 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret") pod "global-pull-secret-syncer-t4dxk" (UID: "5465cf73-cdb9-46e8-ad04-70550bc615a8") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:49:55.407941 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:55.407705 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs podName:5d43201c-5806-4536-8da3-8ad8fa286eac nodeName:}" failed. No retries permitted until 2026-07-22 11:50:11.407689925 +0000 UTC m=+34.156901510 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs") pod "network-metrics-daemon-vk2qp" (UID: "5d43201c-5806-4536-8da3-8ad8fa286eac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:49:55.407941 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:55.407722 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn podName:07c0d7ac-5517-4327-a6cc-4c7c105ac449 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:11.407712426 +0000 UTC m=+34.156924011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zsckn" (UniqueName: "kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn") pod "network-check-target-78b62" (UID: "07c0d7ac-5517-4327-a6cc-4c7c105ac449") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:49:55.953312 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:55.953277 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:55.953499 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:55.953286 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:55.953499 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:55.953410 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:49:55.953610 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:55.953496 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:49:56.954004 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:56.953967 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:56.954004 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:56.954000 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:56.954518 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:56.954095 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:49:56.954518 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:56.954247 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:49:57.955043 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:57.955004 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:57.955456 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:57.955158 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:49:58.953057 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:58.953028 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:49:58.953263 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:58.953026 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:49:58.953263 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:58.953171 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:49:58.953263 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:58.953231 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:49:59.954028 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:49:59.953743 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:49:59.954632 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:49:59.954140 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:50:00.075493 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.075366 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dc7km" event={"ID":"d49ef242-9548-4f15-a5b6-b3278c34e011","Type":"ContainerStarted","Data":"b8ccb63228beb5b0fe8e6e479c9622bc760d0de7550966ac8a8d86fea7ec53bb"} Jul 22 11:50:00.076629 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.076603 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4s464" event={"ID":"57933426-ecf3-46ae-87b9-9a9f8329462e","Type":"ContainerStarted","Data":"6dc00b4e394ba8c1a932fef1ba100fb224024940fb019279cc7b61a92f6a8968"} Jul 22 11:50:00.078035 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.078012 2555 generic.go:356] "Generic (PLEG): container finished" podID="3cb469cd-363d-4822-906b-5a00b7c0e230" containerID="8696d3e7fa614093523c1f1ad192f887b03dced868bb9a5e6ac6ef357659faea" exitCode=0 Jul 22 11:50:00.078158 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.078063 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mfbk" event={"ID":"3cb469cd-363d-4822-906b-5a00b7c0e230","Type":"ContainerDied","Data":"8696d3e7fa614093523c1f1ad192f887b03dced868bb9a5e6ac6ef357659faea"} Jul 22 11:50:00.079588 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.079569 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrmnr" event={"ID":"20f96103-fe3f-4f0f-8f59-351a00064147","Type":"ContainerStarted","Data":"f9cb9e6a06e11a161f21c1b0fa98a0a7ab9b0c490c94f6e8662eb8106dc99e1d"} Jul 22 11:50:00.081141 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.081097 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" event={"ID":"83fbbb08-6d3f-455d-ac19-13386bb78ed5","Type":"ContainerStarted","Data":"967a2c0b1ffab5d076aa3241fef51b1941c80de5161d4c235b16c373f8ce3b18"} Jul 22 11:50:00.082611 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.082593 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" event={"ID":"823c16cc-5e05-42e0-a7e4-bb57adbb2314","Type":"ContainerStarted","Data":"43a2e5ed6e5011515848914eeb3c79057cfb400b569c362a2338e05e2adc5e7d"} Jul 22 11:50:00.084470 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.084453 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vclwc" event={"ID":"41bf461d-75a8-4bed-8d22-bb7cdd9bf266","Type":"ContainerStarted","Data":"91318dba48240805367acb2ab440903002041d341f2e337b5a82568b4c91309e"} Jul 22 11:50:00.087123 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.087088 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" event={"ID":"159c1c65-a08a-4f75-8a16-f0469ae1428f","Type":"ContainerStarted","Data":"51228a03f922f2e9364de631fce25ae96a58cad6f3123793af27af670cc36a2b"} Jul 22 11:50:00.087198 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.087156 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" event={"ID":"159c1c65-a08a-4f75-8a16-f0469ae1428f","Type":"ContainerStarted","Data":"ea899dff3250212ad172fea975ea1a9d66fad8615a7b86abb20f35a266ae9e6d"} Jul 22 11:50:00.087198 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.087186 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" event={"ID":"159c1c65-a08a-4f75-8a16-f0469ae1428f","Type":"ContainerStarted","Data":"afb96244aa1f6d2724cfe0531875c1ba0eca12a62dbbcca012f7c5827e5da8e0"} Jul 22 11:50:00.087291 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.087214 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" event={"ID":"159c1c65-a08a-4f75-8a16-f0469ae1428f","Type":"ContainerStarted","Data":"f7d984f56425333bacdba230619e18dd09191c14472036e849634f9cb8c40cd0"} Jul 22 11:50:00.087291 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.087244 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" event={"ID":"159c1c65-a08a-4f75-8a16-f0469ae1428f","Type":"ContainerStarted","Data":"f0df89f51c4db9dad905faec1443f9f289068af6b3a353b5d96cd663d60efeb9"} Jul 22 11:50:00.101254 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.101167 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.ec2.internal" podStartSLOduration=21.101143629 podStartE2EDuration="21.101143629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:49:43.05854796 +0000 UTC m=+5.807759560" watchObservedRunningTime="2026-07-22 11:50:00.101143629 +0000 UTC m=+22.850355230" Jul 22 11:50:00.101520 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.101500 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dc7km" podStartSLOduration=3.358554851 podStartE2EDuration="22.101492671s" totalImagesPullingTime="18.74293782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:00.101028115 +0000 UTC m=+22.850239718" watchObservedRunningTime="2026-07-22 11:50:00.101492671 +0000 UTC m=+22.850704273" Jul 22 11:50:00.151526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.151476 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4s464" podStartSLOduration=4.468192389 podStartE2EDuration="23.151460717s" totalImagesPullingTime="18.683268328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:00.120489836 +0000 UTC m=+22.869701449" watchObservedRunningTime="2026-07-22 11:50:00.151460717 +0000 UTC m=+22.900672319" Jul 22 11:50:00.173959 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.173902 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qrmnr" podStartSLOduration=3.496311285 podStartE2EDuration="22.17388769s" totalImagesPullingTime="18.677576405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:00.172327356 +0000 UTC m=+22.921538959" watchObservedRunningTime="2026-07-22 11:50:00.17388769 +0000 UTC m=+22.923099291" Jul 22 11:50:00.196071 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.196019 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mxvfx" podStartSLOduration=3.512843416 podStartE2EDuration="22.196003914s" totalImagesPullingTime="18.683160498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:00.19582035 +0000 UTC m=+22.945031955" watchObservedRunningTime="2026-07-22 11:50:00.196003914 +0000 UTC m=+22.945215516" Jul 22 11:50:00.221156 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.221074 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vclwc" podStartSLOduration=3.54151957 podStartE2EDuration="22.22106046s" totalImagesPullingTime="18.67954089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:00.220505492 +0000 UTC m=+22.969717116" watchObservedRunningTime="2026-07-22 11:50:00.22106046 +0000 UTC m=+22.970272062" Jul 22 11:50:00.953384 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.953358 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:50:00.953384 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.953380 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:50:00.953612 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:00.953495 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:50:00.953690 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:00.953666 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:50:00.966435 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.966411 2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 11:50:00.975324 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.975299 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4s464" Jul 22 11:50:00.975843 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:00.975827 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4s464" Jul 22 11:50:01.091181 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:01.091141 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" event={"ID":"83fbbb08-6d3f-455d-ac19-13386bb78ed5","Type":"ContainerStarted","Data":"7953bfd6703223dea94ab135c76cfa0d51a512506a84f85f89651889396fc3ee"} Jul 22 11:50:01.096578 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:01.096545 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" event={"ID":"159c1c65-a08a-4f75-8a16-f0469ae1428f","Type":"ContainerStarted","Data":"0836225463c419648b33ba7f568f797e8cd9bb49465b958abd7958c7366b2207"} Jul 22 11:50:01.098278 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:01.098207 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z8gks" event={"ID":"bff6fe23-8563-41c9-a23b-4e62b98ffa1f","Type":"ContainerStarted","Data":"f19bcc5b16c1135323e512261cec80e7ba2209ae1fe447ef4f803309e3f37397"} Jul 22 11:50:01.116639 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:01.116580 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z8gks" podStartSLOduration=5.411268012 podStartE2EDuration="24.116563337s" totalImagesPullingTime="18.705295325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:01.116398065 +0000 UTC m=+23.865609669" watchObservedRunningTime="2026-07-22 11:50:01.116563337 +0000 UTC m=+23.865774939" Jul 22 11:50:01.892063 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:01.891945 2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T11:50:00.966431502Z","UUID":"761b429b-438a-4d3d-adcc-01976b18e2a3","Handler":null,"Name":"","Endpoint":""} Jul 22 11:50:01.893890 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:01.893864 2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 11:50:01.894093 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:01.893914 2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 11:50:01.953871 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:01.953832 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:50:01.954046 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:01.953987 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:50:02.100466 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:02.100437 2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 11:50:02.953361 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:02.953177 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:50:02.953541 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:02.953175 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:50:02.953541 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:02.953457 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:50:02.953634 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:02.953541 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:50:03.110045 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:03.110006 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" event={"ID":"159c1c65-a08a-4f75-8a16-f0469ae1428f","Type":"ContainerStarted","Data":"c3d4ca60402c834f4a595c7bec75fb94d6c86dfc3ab6e5d5af8e83b7cee3469e"} Jul 22 11:50:03.112499 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:03.112471 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" event={"ID":"83fbbb08-6d3f-455d-ac19-13386bb78ed5","Type":"ContainerStarted","Data":"b0ebb229656f4d3e90973b1dcd90d70725e76ac28fbcd31157162ebdb30f6c43"} Jul 22 11:50:03.137428 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:03.137355 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lws4n" podStartSLOduration=4.05311149 podStartE2EDuration="25.137336895s" totalImagesPullingTime="21.084225405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:49:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:03.137124955 +0000 UTC m=+25.886336557" watchObservedRunningTime="2026-07-22 11:50:03.137336895 +0000 UTC m=+25.886548497" Jul 22 11:50:03.533922 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:03.533886 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4s464" Jul 22 11:50:03.534151 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:03.534037 2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 11:50:03.534547 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:03.534524 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4s464" Jul 22 11:50:03.953168 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:03.953138 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:50:03.953330 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:03.953258 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:50:04.953526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:04.953347 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:50:04.954297 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:04.953391 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:50:04.954297 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:04.953609 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:50:04.954297 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:04.953699 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:50:05.119233 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:05.119201 2555 generic.go:356] "Generic (PLEG): container finished" podID="3cb469cd-363d-4822-906b-5a00b7c0e230" containerID="7931ac6e91b70b5161d2c7d107807e3a2d5824451f372b8f0b9b2fc3d9567493" exitCode=0 Jul 22 11:50:05.119395 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:05.119257 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mfbk" event={"ID":"3cb469cd-363d-4822-906b-5a00b7c0e230","Type":"ContainerDied","Data":"7931ac6e91b70b5161d2c7d107807e3a2d5824451f372b8f0b9b2fc3d9567493"} Jul 22 11:50:05.122990 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:05.122969 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" event={"ID":"159c1c65-a08a-4f75-8a16-f0469ae1428f","Type":"ContainerStarted","Data":"63b415be184514bee9672059ea998093721e858b261c0892511c3132225d8704"} Jul 22 11:50:05.123309 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:05.123293 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:50:05.139858 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:05.139829 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:50:05.954046 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:05.954019 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:50:05.954533 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:05.954137 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:50:06.125499 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:06.125473 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:50:06.125680 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:06.125544 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:50:06.140282 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:06.140257 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:50:06.176622 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:06.176574 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" podStartSLOduration=9.427603659 podStartE2EDuration="28.176559823s" totalImagesPullingTime="18.748956164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:05.20027936 +0000 UTC m=+27.949490955" watchObservedRunningTime="2026-07-22 11:50:06.176559823 +0000 UTC m=+28.925771426" Jul 22 11:50:06.808678 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:06.808640 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t4dxk"] Jul 22 11:50:06.808867 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:06.808757 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:50:06.808911 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:06.808870 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:50:06.811345 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:06.811318 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-78b62"] Jul 22 11:50:06.811460 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:06.811436 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:50:06.811532 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:06.811515 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:50:06.824248 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:06.824186 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vk2qp"] Jul 22 11:50:06.824351 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:06.824292 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:50:06.824443 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:06.824405 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:50:07.129720 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:07.129686 2555 generic.go:356] "Generic (PLEG): container finished" podID="3cb469cd-363d-4822-906b-5a00b7c0e230" containerID="ac5a965bc0c9b7b0c8f5222a0bdead179371f712ce336a1cb177d8e51f2c18fa" exitCode=0 Jul 22 11:50:07.130221 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:07.129747 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mfbk" event={"ID":"3cb469cd-363d-4822-906b-5a00b7c0e230","Type":"ContainerDied","Data":"ac5a965bc0c9b7b0c8f5222a0bdead179371f712ce336a1cb177d8e51f2c18fa"} Jul 22 11:50:08.275251 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:08.275056 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:08.376787 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:08.376680 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:08.953320 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:08.953285 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:50:08.953508 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:08.953402 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:50:08.953508 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:08.953413 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:50:08.953508 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:08.953433 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:50:08.953668 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:08.953522 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:50:08.953668 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:08.953588 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:50:09.124319 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:09.124279 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:09.136019 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:09.135992 2555 generic.go:356] "Generic (PLEG): container finished" podID="3cb469cd-363d-4822-906b-5a00b7c0e230" containerID="337fc64e7dfb780d9af514fc2ddc38ab651f7bca39bc7ac752aadefafe137324" exitCode=0 Jul 22 11:50:09.136181 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:09.136050 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mfbk" event={"ID":"3cb469cd-363d-4822-906b-5a00b7c0e230","Type":"ContainerDied","Data":"337fc64e7dfb780d9af514fc2ddc38ab651f7bca39bc7ac752aadefafe137324"} Jul 22 11:50:09.225379 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:09.225289 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:10.665335 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:10.665204 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:10.783006 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:10.782966 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:10.889233 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:10.889196 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:10.953619 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:10.953539 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:50:10.953789 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:10.953554 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:50:10.953789 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:10.953659 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dxk" podUID="5465cf73-cdb9-46e8-ad04-70550bc615a8" Jul 22 11:50:10.953789 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:10.953670 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:50:10.953789 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:10.953731 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-78b62" podUID="07c0d7ac-5517-4327-a6cc-4c7c105ac449" Jul 22 11:50:10.953973 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:10.953813 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk2qp" podUID="5d43201c-5806-4536-8da3-8ad8fa286eac" Jul 22 11:50:10.992375 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:10.992332 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:11.420640 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:11.420596 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsckn\" (UniqueName: \"kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn\") pod \"network-check-target-78b62\" (UID: \"07c0d7ac-5517-4327-a6cc-4c7c105ac449\") " pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:50:11.420840 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:11.420651 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs\") pod \"network-metrics-daemon-vk2qp\" (UID: \"5d43201c-5806-4536-8da3-8ad8fa286eac\") " pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:50:11.420840 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:11.420695 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:50:11.420840 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:11.420796 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:50:11.420840 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:11.420818 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:50:11.420840 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:11.420832 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:50:11.420840 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:11.420847 2555 projected.go:196] Error preparing data for projected volume kube-api-access-zsckn for pod openshift-network-diagnostics/network-check-target-78b62: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:50:11.421193 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:11.420807 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:50:11.421193 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:11.420890 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs podName:5d43201c-5806-4536-8da3-8ad8fa286eac nodeName:}" failed. No retries permitted until 2026-07-22 11:50:43.420868944 +0000 UTC m=+66.170080526 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs") pod "network-metrics-daemon-vk2qp" (UID: "5d43201c-5806-4536-8da3-8ad8fa286eac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:50:11.421193 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:11.420912 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn podName:07c0d7ac-5517-4327-a6cc-4c7c105ac449 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:43.42090048 +0000 UTC m=+66.170112066 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zsckn" (UniqueName: "kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn") pod "network-check-target-78b62" (UID: "07c0d7ac-5517-4327-a6cc-4c7c105ac449") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:50:11.421193 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:11.420928 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret podName:5465cf73-cdb9-46e8-ad04-70550bc615a8 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:43.42091959 +0000 UTC m=+66.170131176 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret") pod "global-pull-secret-syncer-t4dxk" (UID: "5465cf73-cdb9-46e8-ad04-70550bc615a8") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:50:11.794526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:11.793489 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:11.968341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:11.968309 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:12.557213 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.557178 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-190.ec2.internal" event="NodeReady" Jul 22 11:50:12.557393 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.557342 2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 11:50:12.598483 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.598448 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:12.636088 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.635923 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59868b474b-zfdvd"] Jul 22 11:50:12.667388 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.667357 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h425l"] Jul 22 11:50:12.667557 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.667495 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.671411 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.671387 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 11:50:12.671411 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.671402 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 11:50:12.671752 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.671735 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f7k7c\"" Jul 22 11:50:12.671952 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.671919 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 11:50:12.681484 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.681451 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bfjgh"] Jul 22 11:50:12.681611 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.681568 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h425l" Jul 22 11:50:12.682778 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.682760 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 11:50:12.685270 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.685252 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2xm6s\"" Jul 22 11:50:12.685784 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.685760 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 11:50:12.685880 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.685823 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 11:50:12.701559 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.701529 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h425l"] Jul 22 11:50:12.701698 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.701575 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59868b474b-zfdvd"] Jul 22 11:50:12.701698 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.701598 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bfjgh"] Jul 22 11:50:12.701698 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.701659 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bfjgh" Jul 22 11:50:12.704366 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.704341 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 11:50:12.704696 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.704676 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wgjkp\"" Jul 22 11:50:12.705315 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.705294 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 11:50:12.707327 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.707306 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 11:50:12.721559 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.721527 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:12.731140 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731090 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-certificates\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.731284 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731144 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-bound-sa-token\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.731284 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731164 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbtd8\" (UniqueName: \"kubernetes.io/projected/4b71e182-de9c-49c1-802b-312441501177-kube-api-access-bbtd8\") pod \"ingress-canary-bfjgh\" (UID: \"4b71e182-de9c-49c1-802b-312441501177\") " pod="openshift-ingress-canary/ingress-canary-bfjgh" Jul 22 11:50:12.731398 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731295 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnzf6\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-kube-api-access-lnzf6\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.731398 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731339 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:12.731398 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731363 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68931960-b129-4f9d-bc5a-a10e10c5757b-tmp-dir\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:12.731398 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731391 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9nn5\" (UniqueName: \"kubernetes.io/projected/68931960-b129-4f9d-bc5a-a10e10c5757b-kube-api-access-w9nn5\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:12.731593 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731450 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c87f5690-c6af-4536-a87f-de51ac6cec05-image-registry-private-configuration\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.731593 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731480 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c87f5690-c6af-4536-a87f-de51ac6cec05-ca-trust-extracted\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.731593 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731513 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c87f5690-c6af-4536-a87f-de51ac6cec05-installation-pull-secrets\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.731593 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731576 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.731755 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731602 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b71e182-de9c-49c1-802b-312441501177-cert\") pod \"ingress-canary-bfjgh\" (UID: \"4b71e182-de9c-49c1-802b-312441501177\") " pod="openshift-ingress-canary/ingress-canary-bfjgh" Jul 22 11:50:12.731755 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731629 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68931960-b129-4f9d-bc5a-a10e10c5757b-config-volume\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:12.731755 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.731665 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c87f5690-c6af-4536-a87f-de51ac6cec05-trusted-ca\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.826436 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.826336 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:12.832194 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832168 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c87f5690-c6af-4536-a87f-de51ac6cec05-installation-pull-secrets\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.832330 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832207 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.832330 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832231 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b71e182-de9c-49c1-802b-312441501177-cert\") pod \"ingress-canary-bfjgh\" (UID: \"4b71e182-de9c-49c1-802b-312441501177\") " pod="openshift-ingress-canary/ingress-canary-bfjgh" Jul 22 11:50:12.832330 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832247 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68931960-b129-4f9d-bc5a-a10e10c5757b-config-volume\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:12.832330 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832263 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c87f5690-c6af-4536-a87f-de51ac6cec05-trusted-ca\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.832330 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832288 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-certificates\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.832330 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832310 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-bound-sa-token\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.832330 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832330 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbtd8\" (UniqueName: \"kubernetes.io/projected/4b71e182-de9c-49c1-802b-312441501177-kube-api-access-bbtd8\") pod \"ingress-canary-bfjgh\" (UID: \"4b71e182-de9c-49c1-802b-312441501177\") " pod="openshift-ingress-canary/ingress-canary-bfjgh" Jul 22 11:50:12.832637 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:12.832332 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 11:50:12.832637 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:12.832355 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59868b474b-zfdvd: secret "image-registry-tls" not found Jul 22 11:50:12.832637 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832390 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnzf6\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-kube-api-access-lnzf6\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.832637 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:12.832418 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls podName:c87f5690-c6af-4536-a87f-de51ac6cec05 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:13.332396627 +0000 UTC m=+36.081608209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls") pod "image-registry-59868b474b-zfdvd" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05") : secret "image-registry-tls" not found Jul 22 11:50:12.832637 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832466 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:12.832637 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832490 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68931960-b129-4f9d-bc5a-a10e10c5757b-tmp-dir\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:12.832637 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832518 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9nn5\" (UniqueName: \"kubernetes.io/projected/68931960-b129-4f9d-bc5a-a10e10c5757b-kube-api-access-w9nn5\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:12.832637 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832563 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c87f5690-c6af-4536-a87f-de51ac6cec05-image-registry-private-configuration\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.832637 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832584 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c87f5690-c6af-4536-a87f-de51ac6cec05-ca-trust-extracted\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.833077 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832879 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c87f5690-c6af-4536-a87f-de51ac6cec05-ca-trust-extracted\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.833077 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.832917 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68931960-b129-4f9d-bc5a-a10e10c5757b-config-volume\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:12.833077 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:12.833017 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:50:12.833077 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:12.833077 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls podName:68931960-b129-4f9d-bc5a-a10e10c5757b nodeName:}" failed. No retries permitted until 2026-07-22 11:50:13.333060494 +0000 UTC m=+36.082272083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls") pod "dns-default-h425l" (UID: "68931960-b129-4f9d-bc5a-a10e10c5757b") : secret "dns-default-metrics-tls" not found Jul 22 11:50:12.833639 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.833611 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-certificates\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.833761 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.833616 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c87f5690-c6af-4536-a87f-de51ac6cec05-trusted-ca\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.837575 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.837546 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c87f5690-c6af-4536-a87f-de51ac6cec05-image-registry-private-configuration\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.837575 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.837560 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b71e182-de9c-49c1-802b-312441501177-cert\") pod \"ingress-canary-bfjgh\" (UID: \"4b71e182-de9c-49c1-802b-312441501177\") " pod="openshift-ingress-canary/ingress-canary-bfjgh" Jul 22 11:50:12.837892 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.837867 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c87f5690-c6af-4536-a87f-de51ac6cec05-installation-pull-secrets\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.841631 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.841607 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68931960-b129-4f9d-bc5a-a10e10c5757b-tmp-dir\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:12.843139 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.843079 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnzf6\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-kube-api-access-lnzf6\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.843785 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.843763 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbtd8\" (UniqueName: \"kubernetes.io/projected/4b71e182-de9c-49c1-802b-312441501177-kube-api-access-bbtd8\") pod \"ingress-canary-bfjgh\" (UID: \"4b71e182-de9c-49c1-802b-312441501177\") " pod="openshift-ingress-canary/ingress-canary-bfjgh" Jul 22 11:50:12.844099 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.844056 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9nn5\" (UniqueName: \"kubernetes.io/projected/68931960-b129-4f9d-bc5a-a10e10c5757b-kube-api-access-w9nn5\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:12.846144 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.846124 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-bound-sa-token\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:12.925793 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.925751 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:12.953700 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.953661 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:50:12.953869 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.953706 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:50:12.953869 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.953706 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:50:12.957050 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.957027 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 11:50:12.957693 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.957554 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mjncz\"" Jul 22 11:50:12.957693 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.957571 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b6dpz\"" Jul 22 11:50:12.957693 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.957640 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 11:50:12.957909 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.957862 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 11:50:12.957909 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:12.957865 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 11:50:13.011760 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:13.011727 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bfjgh" Jul 22 11:50:13.028585 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:13.028547 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:13.245452 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:13.245196 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bfjgh"] Jul 22 11:50:13.258133 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:50:13.258094 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b71e182_de9c_49c1_802b_312441501177.slice/crio-a6dd66e04189c6a84c53342aa074d7cdcaf242d4ed4c68672021e12db8e4ca74 WatchSource:0}: Error finding container a6dd66e04189c6a84c53342aa074d7cdcaf242d4ed4c68672021e12db8e4ca74: Status 404 returned error can't find the container with id a6dd66e04189c6a84c53342aa074d7cdcaf242d4ed4c68672021e12db8e4ca74 Jul 22 11:50:13.337441 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:13.337417 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:13.337563 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:13.337476 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:13.337601 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:13.337576 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 11:50:13.337601 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:13.337599 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59868b474b-zfdvd: secret "image-registry-tls" not found Jul 22 11:50:13.337667 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:13.337651 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls podName:c87f5690-c6af-4536-a87f-de51ac6cec05 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:14.337635131 +0000 UTC m=+37.086846710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls") pod "image-registry-59868b474b-zfdvd" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05") : secret "image-registry-tls" not found Jul 22 11:50:13.337667 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:13.337574 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:50:13.337731 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:13.337699 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls podName:68931960-b129-4f9d-bc5a-a10e10c5757b nodeName:}" failed. No retries permitted until 2026-07-22 11:50:14.337689182 +0000 UTC m=+37.086900767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls") pod "dns-default-h425l" (UID: "68931960-b129-4f9d-bc5a-a10e10c5757b") : secret "dns-default-metrics-tls" not found Jul 22 11:50:14.149769 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:14.149731 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bfjgh" event={"ID":"4b71e182-de9c-49c1-802b-312441501177","Type":"ContainerStarted","Data":"a6dd66e04189c6a84c53342aa074d7cdcaf242d4ed4c68672021e12db8e4ca74"} Jul 22 11:50:14.152761 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:14.152734 2555 generic.go:356] "Generic (PLEG): container finished" podID="3cb469cd-363d-4822-906b-5a00b7c0e230" containerID="83a13335668cdf2cd13dec95d263e0f71e66ac8adf795642ad6d3671d6193594" exitCode=0 Jul 22 11:50:14.152892 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:14.152800 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mfbk" event={"ID":"3cb469cd-363d-4822-906b-5a00b7c0e230","Type":"ContainerDied","Data":"83a13335668cdf2cd13dec95d263e0f71e66ac8adf795642ad6d3671d6193594"} Jul 22 11:50:14.345968 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:14.345928 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:14.346152 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:14.346006 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:14.346152 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:14.346095 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 11:50:14.346152 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:14.346122 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:50:14.346152 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:14.346128 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59868b474b-zfdvd: secret "image-registry-tls" not found Jul 22 11:50:14.346304 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:14.346188 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls podName:68931960-b129-4f9d-bc5a-a10e10c5757b nodeName:}" failed. No retries permitted until 2026-07-22 11:50:16.346169731 +0000 UTC m=+39.095381316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls") pod "dns-default-h425l" (UID: "68931960-b129-4f9d-bc5a-a10e10c5757b") : secret "dns-default-metrics-tls" not found Jul 22 11:50:14.346304 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:14.346206 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls podName:c87f5690-c6af-4536-a87f-de51ac6cec05 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:16.346196845 +0000 UTC m=+39.095408432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls") pod "image-registry-59868b474b-zfdvd" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05") : secret "image-registry-tls" not found Jul 22 11:50:15.158911 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:15.158872 2555 generic.go:356] "Generic (PLEG): container finished" podID="3cb469cd-363d-4822-906b-5a00b7c0e230" containerID="c7184dde2a0009cd55ede1fd05355bd61a279a0b0421d32e2928215489a2473e" exitCode=0 Jul 22 11:50:15.159327 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:15.158930 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mfbk" event={"ID":"3cb469cd-363d-4822-906b-5a00b7c0e230","Type":"ContainerDied","Data":"c7184dde2a0009cd55ede1fd05355bd61a279a0b0421d32e2928215489a2473e"} Jul 22 11:50:16.162617 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:16.162432 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bfjgh" event={"ID":"4b71e182-de9c-49c1-802b-312441501177","Type":"ContainerStarted","Data":"974c307bbc1dd815398172a613e98049b2eea8ff7429db2e213cc1d54c1eff41"} Jul 22 11:50:16.165562 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:16.165536 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mfbk" event={"ID":"3cb469cd-363d-4822-906b-5a00b7c0e230","Type":"ContainerStarted","Data":"f493628ff1765dc8986c3ec104d2eff88ba2ee74ca982d5bb614426f7df0e51e"} Jul 22 11:50:16.183192 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:16.183126 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bfjgh" podStartSLOduration=2.110677796 podStartE2EDuration="4.183084409s" totalImagesPullingTime="2.072406613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:16.182407816 +0000 UTC m=+38.931619421" watchObservedRunningTime="2026-07-22 11:50:16.183084409 +0000 UTC m=+38.932296012" Jul 22 11:50:16.207562 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:16.207502 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4mfbk" podStartSLOduration=8.834619275 podStartE2EDuration="38.207466861s" totalImagesPullingTime="29.372847586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:49:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:16.205357857 +0000 UTC m=+38.954569459" watchObservedRunningTime="2026-07-22 11:50:16.207466861 +0000 UTC m=+38.956678463" Jul 22 11:50:16.336620 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:16.336580 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:16.362686 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:16.362663 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:16.362794 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:16.362714 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:16.362829 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:16.362805 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 11:50:16.362829 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:16.362806 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:50:16.362891 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:16.362867 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls podName:68931960-b129-4f9d-bc5a-a10e10c5757b nodeName:}" failed. No retries permitted until 2026-07-22 11:50:20.362852008 +0000 UTC m=+43.112063588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls") pod "dns-default-h425l" (UID: "68931960-b129-4f9d-bc5a-a10e10c5757b") : secret "dns-default-metrics-tls" not found Jul 22 11:50:16.362928 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:16.362814 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59868b474b-zfdvd: secret "image-registry-tls" not found Jul 22 11:50:16.362960 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:16.362935 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls podName:c87f5690-c6af-4536-a87f-de51ac6cec05 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:20.362920354 +0000 UTC m=+43.112131940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls") pod "image-registry-59868b474b-zfdvd" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05") : secret "image-registry-tls" not found Jul 22 11:50:16.457140 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:16.457045 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:16.908236 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:16.908204 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:17.014546 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:17.014512 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:18.687371 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:18.687334 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:18.790588 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:18.790550 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:19.022600 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.022531 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pz6s9"] Jul 22 11:50:19.025847 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.025830 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.036254 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.036237 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 11:50:19.036342 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.036271 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8jdv6\"" Jul 22 11:50:19.037085 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.037062 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 11:50:19.037207 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.037062 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 11:50:19.037207 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.037069 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 11:50:19.047817 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.047792 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pz6s9"] Jul 22 11:50:19.079309 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.079282 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-dzsb2"] Jul 22 11:50:19.082548 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.082520 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-dzsb2" Jul 22 11:50:19.082984 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.082953 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:19.084867 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.084849 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae05e4be-c308-457f-aaeb-c47ae8637746-data-volume\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.084948 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.084893 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.085005 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.084953 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae05e4be-c308-457f-aaeb-c47ae8637746-crio-socket\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.085005 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.084980 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.085083 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.085044 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdfkq\" (UniqueName: \"kubernetes.io/projected/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-api-access-bdfkq\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.085457 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.085442 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 22 11:50:19.085533 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.085448 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 22 11:50:19.085956 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.085940 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ql4z2\"" Jul 22 11:50:19.097388 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.097370 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-dzsb2"] Jul 22 11:50:19.170614 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.170578 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:19.185761 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.185735 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h65bs\" (UniqueName: \"kubernetes.io/projected/40e956c0-f111-42f7-879b-f30d70bca3f8-kube-api-access-h65bs\") pod \"migrator-7cc9c7ffd7-dzsb2\" (UID: \"40e956c0-f111-42f7-879b-f30d70bca3f8\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-dzsb2" Jul 22 11:50:19.185898 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.185772 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.185898 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.185830 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae05e4be-c308-457f-aaeb-c47ae8637746-crio-socket\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.185987 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.185910 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.185987 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.185936 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdfkq\" (UniqueName: \"kubernetes.io/projected/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-api-access-bdfkq\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.185987 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:19.185944 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:50:19.186146 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.185997 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae05e4be-c308-457f-aaeb-c47ae8637746-data-volume\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.186146 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:19.186012 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls podName:ae05e4be-c308-457f-aaeb-c47ae8637746 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:19.685992424 +0000 UTC m=+42.435204006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pz6s9" (UID: "ae05e4be-c308-457f-aaeb-c47ae8637746") : secret "insights-runtime-extractor-tls" not found Jul 22 11:50:19.186146 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.186014 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae05e4be-c308-457f-aaeb-c47ae8637746-crio-socket\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.186146 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:19.186033 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:50:19.186146 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:19.186135 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm podName:ae05e4be-c308-457f-aaeb-c47ae8637746 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:19.686094026 +0000 UTC m=+42.435305619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pz6s9" (UID: "ae05e4be-c308-457f-aaeb-c47ae8637746") : configmap "kube-rbac-proxy" not found Jul 22 11:50:19.186343 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.186325 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae05e4be-c308-457f-aaeb-c47ae8637746-data-volume\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.199062 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.199038 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdfkq\" (UniqueName: \"kubernetes.io/projected/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-api-access-bdfkq\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.270117 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.270076 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:19.287425 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.287334 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h65bs\" (UniqueName: \"kubernetes.io/projected/40e956c0-f111-42f7-879b-f30d70bca3f8-kube-api-access-h65bs\") pod \"migrator-7cc9c7ffd7-dzsb2\" (UID: \"40e956c0-f111-42f7-879b-f30d70bca3f8\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-dzsb2" Jul 22 11:50:19.298023 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.297994 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h65bs\" (UniqueName: \"kubernetes.io/projected/40e956c0-f111-42f7-879b-f30d70bca3f8-kube-api-access-h65bs\") pod \"migrator-7cc9c7ffd7-dzsb2\" (UID: \"40e956c0-f111-42f7-879b-f30d70bca3f8\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-dzsb2" Jul 22 11:50:19.371228 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.371194 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:19.391170 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.391137 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-dzsb2" Jul 22 11:50:19.535946 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.535918 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-dzsb2"] Jul 22 11:50:19.538910 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:50:19.538850 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e956c0_f111_42f7_879b_f30d70bca3f8.slice/crio-ddaa55384956ddaad646a576f69b41722c39a0def9d58d11c77b14f0b75c0e2b WatchSource:0}: Error finding container ddaa55384956ddaad646a576f69b41722c39a0def9d58d11c77b14f0b75c0e2b: Status 404 returned error can't find the container with id ddaa55384956ddaad646a576f69b41722c39a0def9d58d11c77b14f0b75c0e2b Jul 22 11:50:19.690365 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.690333 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.690730 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.690425 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:19.690730 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:19.690491 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:50:19.690730 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:19.690540 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:50:19.690730 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:19.690555 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls podName:ae05e4be-c308-457f-aaeb-c47ae8637746 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:20.690539401 +0000 UTC m=+43.439750981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pz6s9" (UID: "ae05e4be-c308-457f-aaeb-c47ae8637746") : secret "insights-runtime-extractor-tls" not found Jul 22 11:50:19.690730 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:19.690591 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm podName:ae05e4be-c308-457f-aaeb-c47ae8637746 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:20.690576316 +0000 UTC m=+43.439787899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pz6s9" (UID: "ae05e4be-c308-457f-aaeb-c47ae8637746") : configmap "kube-rbac-proxy" not found Jul 22 11:50:19.769494 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.769460 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:19.869820 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:19.869786 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:20.176503 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:20.176416 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-dzsb2" event={"ID":"40e956c0-f111-42f7-879b-f30d70bca3f8","Type":"ContainerStarted","Data":"ddaa55384956ddaad646a576f69b41722c39a0def9d58d11c77b14f0b75c0e2b"} Jul 22 11:50:20.396872 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:20.396834 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:20.397037 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:20.396931 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:20.397037 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:20.397000 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:50:20.397143 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:20.397062 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls podName:68931960-b129-4f9d-bc5a-a10e10c5757b nodeName:}" failed. No retries permitted until 2026-07-22 11:50:28.39704756 +0000 UTC m=+51.146259140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls") pod "dns-default-h425l" (UID: "68931960-b129-4f9d-bc5a-a10e10c5757b") : secret "dns-default-metrics-tls" not found Jul 22 11:50:20.397143 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:20.397068 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 11:50:20.397143 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:20.397087 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59868b474b-zfdvd: secret "image-registry-tls" not found Jul 22 11:50:20.397254 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:20.397146 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls podName:c87f5690-c6af-4536-a87f-de51ac6cec05 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:28.397133535 +0000 UTC m=+51.146345115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls") pod "image-registry-59868b474b-zfdvd" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05") : secret "image-registry-tls" not found Jul 22 11:50:20.699379 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:20.699338 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:20.699747 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:20.699400 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:20.699747 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:20.699493 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:50:20.699747 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:20.699561 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls podName:ae05e4be-c308-457f-aaeb-c47ae8637746 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:22.699543439 +0000 UTC m=+45.448755019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pz6s9" (UID: "ae05e4be-c308-457f-aaeb-c47ae8637746") : secret "insights-runtime-extractor-tls" not found Jul 22 11:50:20.699747 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:20.699496 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:50:20.699747 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:20.699594 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm podName:ae05e4be-c308-457f-aaeb-c47ae8637746 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:22.699588468 +0000 UTC m=+45.448800048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pz6s9" (UID: "ae05e4be-c308-457f-aaeb-c47ae8637746") : configmap "kube-rbac-proxy" not found Jul 22 11:50:20.795212 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:20.795189 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vclwc_41bf461d-75a8-4bed-8d22-bb7cdd9bf266/dns-node-resolver/0.log" Jul 22 11:50:21.179960 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.179932 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-dzsb2" event={"ID":"40e956c0-f111-42f7-879b-f30d70bca3f8","Type":"ContainerStarted","Data":"66376503ba30fc7ea5fbdc0862e0947078df035359d2d73451b322e7ef8c3548"} Jul 22 11:50:21.379443 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.379406 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-s6mrh"] Jul 22 11:50:21.400386 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.400354 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-s6mrh" Jul 22 11:50:21.404820 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.404758 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6t2m2\"" Jul 22 11:50:21.405625 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.405598 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 22 11:50:21.405625 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.405647 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 22 11:50:21.405985 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.405962 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 22 11:50:21.406119 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.406001 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 22 11:50:21.406536 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.406486 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-s6mrh"] Jul 22 11:50:21.486282 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.486247 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:21.506213 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.506184 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrl8d\" (UniqueName: \"kubernetes.io/projected/54d0b989-ea10-461b-a7f2-191ec0ec0f5d-kube-api-access-xrl8d\") pod \"service-ca-58598695f7-s6mrh\" (UID: \"54d0b989-ea10-461b-a7f2-191ec0ec0f5d\") " pod="openshift-service-ca/service-ca-58598695f7-s6mrh" Jul 22 11:50:21.506361 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.506248 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/54d0b989-ea10-461b-a7f2-191ec0ec0f5d-signing-cabundle\") pod \"service-ca-58598695f7-s6mrh\" (UID: \"54d0b989-ea10-461b-a7f2-191ec0ec0f5d\") " pod="openshift-service-ca/service-ca-58598695f7-s6mrh" Jul 22 11:50:21.506361 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.506322 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/54d0b989-ea10-461b-a7f2-191ec0ec0f5d-signing-key\") pod \"service-ca-58598695f7-s6mrh\" (UID: \"54d0b989-ea10-461b-a7f2-191ec0ec0f5d\") " pod="openshift-service-ca/service-ca-58598695f7-s6mrh" Jul 22 11:50:21.589990 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.589953 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:21.607047 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.607010 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrl8d\" (UniqueName: \"kubernetes.io/projected/54d0b989-ea10-461b-a7f2-191ec0ec0f5d-kube-api-access-xrl8d\") pod \"service-ca-58598695f7-s6mrh\" (UID: \"54d0b989-ea10-461b-a7f2-191ec0ec0f5d\") " pod="openshift-service-ca/service-ca-58598695f7-s6mrh" Jul 22 11:50:21.607182 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.607090 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/54d0b989-ea10-461b-a7f2-191ec0ec0f5d-signing-cabundle\") pod \"service-ca-58598695f7-s6mrh\" (UID: \"54d0b989-ea10-461b-a7f2-191ec0ec0f5d\") " pod="openshift-service-ca/service-ca-58598695f7-s6mrh" Jul 22 11:50:21.607182 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.607145 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/54d0b989-ea10-461b-a7f2-191ec0ec0f5d-signing-key\") pod \"service-ca-58598695f7-s6mrh\" (UID: \"54d0b989-ea10-461b-a7f2-191ec0ec0f5d\") " pod="openshift-service-ca/service-ca-58598695f7-s6mrh" Jul 22 11:50:21.607717 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.607691 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/54d0b989-ea10-461b-a7f2-191ec0ec0f5d-signing-cabundle\") pod \"service-ca-58598695f7-s6mrh\" (UID: \"54d0b989-ea10-461b-a7f2-191ec0ec0f5d\") " pod="openshift-service-ca/service-ca-58598695f7-s6mrh" Jul 22 11:50:21.609518 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.609498 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/54d0b989-ea10-461b-a7f2-191ec0ec0f5d-signing-key\") pod \"service-ca-58598695f7-s6mrh\" (UID: \"54d0b989-ea10-461b-a7f2-191ec0ec0f5d\") " pod="openshift-service-ca/service-ca-58598695f7-s6mrh" Jul 22 11:50:21.616926 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.616900 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrl8d\" (UniqueName: \"kubernetes.io/projected/54d0b989-ea10-461b-a7f2-191ec0ec0f5d-kube-api-access-xrl8d\") pod \"service-ca-58598695f7-s6mrh\" (UID: \"54d0b989-ea10-461b-a7f2-191ec0ec0f5d\") " pod="openshift-service-ca/service-ca-58598695f7-s6mrh" Jul 22 11:50:21.713681 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.713594 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-s6mrh" Jul 22 11:50:21.801725 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.801702 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrmnr_20f96103-fe3f-4f0f-8f59-351a00064147/node-ca/0.log" Jul 22 11:50:21.880755 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:21.880722 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-s6mrh"] Jul 22 11:50:21.883993 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:50:21.883962 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d0b989_ea10_461b_a7f2_191ec0ec0f5d.slice/crio-de2837eeef26e9bfef7ff10b627a19b1703b3304d0693b4c811088df6d7a2a22 WatchSource:0}: Error finding container de2837eeef26e9bfef7ff10b627a19b1703b3304d0693b4c811088df6d7a2a22: Status 404 returned error can't find the container with id de2837eeef26e9bfef7ff10b627a19b1703b3304d0693b4c811088df6d7a2a22 Jul 22 11:50:22.184422 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:22.184391 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-dzsb2" event={"ID":"40e956c0-f111-42f7-879b-f30d70bca3f8","Type":"ContainerStarted","Data":"47cb2a1889a20abbb4044e1d4f90317350e1940da51f3a6c5b572be28d26ff14"} Jul 22 11:50:22.185544 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:22.185521 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-s6mrh" event={"ID":"54d0b989-ea10-461b-a7f2-191ec0ec0f5d","Type":"ContainerStarted","Data":"de2837eeef26e9bfef7ff10b627a19b1703b3304d0693b4c811088df6d7a2a22"} Jul 22 11:50:22.196856 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:22.196835 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bfjgh_4b71e182-de9c-49c1-802b-312441501177/serve-healthcheck-canary/0.log" Jul 22 11:50:22.208275 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:22.208165 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-dzsb2" podStartSLOduration=1.705179388 podStartE2EDuration="3.208147159s" totalImagesPullingTime="1.502967771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:22.207663025 +0000 UTC m=+44.956874627" watchObservedRunningTime="2026-07-22 11:50:22.208147159 +0000 UTC m=+44.957358762" Jul 22 11:50:22.684845 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:22.684802 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:22.716782 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:22.716741 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:22.717252 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:22.716880 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:50:22.717252 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:22.716967 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm podName:ae05e4be-c308-457f-aaeb-c47ae8637746 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:26.71694532 +0000 UTC m=+49.466156901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pz6s9" (UID: "ae05e4be-c308-457f-aaeb-c47ae8637746") : configmap "kube-rbac-proxy" not found Jul 22 11:50:22.717252 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:22.717007 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:22.717252 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:22.717101 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:50:22.717252 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:22.717177 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls podName:ae05e4be-c308-457f-aaeb-c47ae8637746 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:26.717159727 +0000 UTC m=+49.466371322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pz6s9" (UID: "ae05e4be-c308-457f-aaeb-c47ae8637746") : secret "insights-runtime-extractor-tls" not found Jul 22 11:50:24.192838 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:24.192800 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-s6mrh" event={"ID":"54d0b989-ea10-461b-a7f2-191ec0ec0f5d","Type":"ContainerStarted","Data":"5f787f0fa8c4dbd338317da19700ce5779b7c6eb94bcb927e38c771bed48582b"} Jul 22 11:50:24.216011 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:24.215958 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-s6mrh" podStartSLOduration=1.057543541 podStartE2EDuration="3.215942477s" totalImagesPullingTime="2.158398936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:24.214838444 +0000 UTC m=+46.964050044" watchObservedRunningTime="2026-07-22 11:50:24.215942477 +0000 UTC m=+46.965154118" Jul 22 11:50:24.909750 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:24.909708 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:25.007967 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:25.007933 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:25.194869 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:25.194784 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:25.295869 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:25.295832 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:25.411010 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:25.410961 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:25.515535 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:25.515416 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:25.616351 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:25.616303 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:25.696246 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:25.696206 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:25.787808 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:25.787713 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:26.208336 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:26.208301 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:26.755209 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:26.755177 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:26.755417 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:26.755394 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:26.755525 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:26.755512 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:50:26.755580 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:26.755571 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls podName:ae05e4be-c308-457f-aaeb-c47ae8637746 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:34.755556978 +0000 UTC m=+57.504768561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pz6s9" (UID: "ae05e4be-c308-457f-aaeb-c47ae8637746") : secret "insights-runtime-extractor-tls" not found Jul 22 11:50:26.755854 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:26.755835 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae05e4be-c308-457f-aaeb-c47ae8637746-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:26.791045 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:26.791008 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:27.098159 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:27.098129 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:27.195081 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:27.195047 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:27.296955 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:27.296923 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:27.395578 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:27.395480 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:27.591145 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:27.591098 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:27.798196 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:27.798088 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:28.198044 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:28.197983 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:28.377100 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:28.377056 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:28.466298 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:28.466226 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:28.466298 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:28.466293 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:28.466471 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:28.466366 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 11:50:28.466471 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:28.466382 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59868b474b-zfdvd: secret "image-registry-tls" not found Jul 22 11:50:28.466471 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:50:28.466448 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls podName:c87f5690-c6af-4536-a87f-de51ac6cec05 nodeName:}" failed. No retries permitted until 2026-07-22 11:50:44.466431293 +0000 UTC m=+67.215642874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls") pod "image-registry-59868b474b-zfdvd" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05") : secret "image-registry-tls" not found Jul 22 11:50:28.468617 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:28.468598 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68931960-b129-4f9d-bc5a-a10e10c5757b-metrics-tls\") pod \"dns-default-h425l\" (UID: \"68931960-b129-4f9d-bc5a-a10e10c5757b\") " pod="openshift-dns/dns-default-h425l" Jul 22 11:50:28.589942 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:28.589908 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:28.592814 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:28.592794 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h425l" Jul 22 11:50:28.734639 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:28.734565 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h425l"] Jul 22 11:50:28.737886 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:50:28.737851 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68931960_b129_4f9d_bc5a_a10e10c5757b.slice/crio-067b0658ae96f515d25b0b3f25c7d501e0bd1b04919a6dd8b6354112b2c79398 WatchSource:0}: Error finding container 067b0658ae96f515d25b0b3f25c7d501e0bd1b04919a6dd8b6354112b2c79398: Status 404 returned error can't find the container with id 067b0658ae96f515d25b0b3f25c7d501e0bd1b04919a6dd8b6354112b2c79398 Jul 22 11:50:28.817995 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:28.817963 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:28.919697 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:28.919655 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:29.207615 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:29.207581 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h425l" event={"ID":"68931960-b129-4f9d-bc5a-a10e10c5757b","Type":"ContainerStarted","Data":"067b0658ae96f515d25b0b3f25c7d501e0bd1b04919a6dd8b6354112b2c79398"} Jul 22 11:50:29.396287 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:29.396255 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:29.589672 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:29.589625 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:29.654310 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:29.654225 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:29.948074 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:29.947983 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:30.045393 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:30.045246 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:30.146776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:30.146739 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:30.248496 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:30.248402 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:30.328516 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:30.328485 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:30.590213 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:30.590095 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:30.689277 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:30.689240 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:30.863603 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:30.863517 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:30.963467 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:30.963407 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:31.107463 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:31.107436 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:31.217822 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:31.217713 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:31.649333 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:31.649296 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:31.853142 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:31.853089 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:32.175674 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:32.175578 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:32.217866 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:32.217831 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h425l" event={"ID":"68931960-b129-4f9d-bc5a-a10e10c5757b","Type":"ContainerStarted","Data":"a21234eda2b5d10b8f29771c5cf5242c4d541eaef3a2eed900228e3e040b5e67"} Jul 22 11:50:32.218037 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:32.217884 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h425l" event={"ID":"68931960-b129-4f9d-bc5a-a10e10c5757b","Type":"ContainerStarted","Data":"8b7c063f86f19d9ab1ad2b2dbb735a3151130af4b81794dba14d75cdcd8ef002"} Jul 22 11:50:32.218037 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:32.217964 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h425l" Jul 22 11:50:32.244917 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:32.244868 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h425l" podStartSLOduration=17.726800654 podStartE2EDuration="20.244853522s" totalImagesPullingTime="2.518052868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:32.243448932 +0000 UTC m=+54.992660534" watchObservedRunningTime="2026-07-22 11:50:32.244853522 +0000 UTC m=+54.994065123" Jul 22 11:50:32.284165 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:32.284132 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:32.639629 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:32.639596 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:32.802496 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:32.802466 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:33.249368 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:33.249324 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:33.626539 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:33.626511 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:34.032490 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:34.032401 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:34.208473 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:34.208439 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:34.645768 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:34.645733 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:34.818904 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:34.818865 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:34.821063 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:34.821033 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae05e4be-c308-457f-aaeb-c47ae8637746-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz6s9\" (UID: \"ae05e4be-c308-457f-aaeb-c47ae8637746\") " pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:34.935065 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:34.934984 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pz6s9" Jul 22 11:50:35.078424 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:35.078387 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pz6s9"] Jul 22 11:50:35.081809 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:50:35.081782 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae05e4be_c308_457f_aaeb_c47ae8637746.slice/crio-5407bb762621aaae7a67e3f6ceae67246165d85ad7fbd1f1e27b362dc9498a74 WatchSource:0}: Error finding container 5407bb762621aaae7a67e3f6ceae67246165d85ad7fbd1f1e27b362dc9498a74: Status 404 returned error can't find the container with id 5407bb762621aaae7a67e3f6ceae67246165d85ad7fbd1f1e27b362dc9498a74 Jul 22 11:50:35.230914 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:35.230825 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pz6s9" event={"ID":"ae05e4be-c308-457f-aaeb-c47ae8637746","Type":"ContainerStarted","Data":"5407bb762621aaae7a67e3f6ceae67246165d85ad7fbd1f1e27b362dc9498a74"} Jul 22 11:50:35.424422 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:35.424384 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:35.597773 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:35.597737 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:35.884880 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:35.884794 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:36.015316 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:36.015250 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:37.259232 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:37.259202 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:38.144495 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:38.144460 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6mthh" Jul 22 11:50:40.634466 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:40.634432 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:40.735616 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:40.735584 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:41.253896 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:41.253863 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pz6s9" event={"ID":"ae05e4be-c308-457f-aaeb-c47ae8637746","Type":"ContainerStarted","Data":"cbd3d7487ae6e8e1b5f145dd68244f6ac495efbf851e4c464ec3be4da2c59c6a"} Jul 22 11:50:42.065426 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:42.065388 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:42.222649 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:42.222616 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h425l" Jul 22 11:50:43.496866 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.496823 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:50:43.497337 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.496931 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsckn\" (UniqueName: \"kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn\") pod \"network-check-target-78b62\" (UID: \"07c0d7ac-5517-4327-a6cc-4c7c105ac449\") " pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:50:43.497337 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.496961 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs\") pod \"network-metrics-daemon-vk2qp\" (UID: \"5d43201c-5806-4536-8da3-8ad8fa286eac\") " pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:50:43.499889 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.499857 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 11:50:43.500874 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.500850 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 11:50:43.500986 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.500896 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 11:50:43.509412 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.509390 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 11:50:43.509563 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.509539 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5465cf73-cdb9-46e8-ad04-70550bc615a8-original-pull-secret\") pod \"global-pull-secret-syncer-t4dxk\" (UID: \"5465cf73-cdb9-46e8-ad04-70550bc615a8\") " pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:50:43.509678 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.509559 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d43201c-5806-4536-8da3-8ad8fa286eac-metrics-certs\") pod \"network-metrics-daemon-vk2qp\" (UID: \"5d43201c-5806-4536-8da3-8ad8fa286eac\") " pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:50:43.519783 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.519759 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsckn\" (UniqueName: \"kubernetes.io/projected/07c0d7ac-5517-4327-a6cc-4c7c105ac449-kube-api-access-zsckn\") pod \"network-check-target-78b62\" (UID: \"07c0d7ac-5517-4327-a6cc-4c7c105ac449\") " pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:50:43.565709 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.565667 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dxk" Jul 22 11:50:43.575532 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.575496 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mjncz\"" Jul 22 11:50:43.581447 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.581428 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b6dpz\"" Jul 22 11:50:43.583280 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.583261 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk2qp" Jul 22 11:50:43.589256 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.589232 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:50:43.862767 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:43.862740 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vk2qp"] Jul 22 11:50:43.865702 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:50:43.865675 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d43201c_5806_4536_8da3_8ad8fa286eac.slice/crio-e2566be96605fd88c2573a2be5fa5e78d29ae8f20580ade4a3f3027ef5acfb69 WatchSource:0}: Error finding container e2566be96605fd88c2573a2be5fa5e78d29ae8f20580ade4a3f3027ef5acfb69: Status 404 returned error can't find the container with id e2566be96605fd88c2573a2be5fa5e78d29ae8f20580ade4a3f3027ef5acfb69 Jul 22 11:50:44.090192 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:44.090158 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-78b62"] Jul 22 11:50:44.092815 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:44.092789 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t4dxk"] Jul 22 11:50:44.093678 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:50:44.093652 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c0d7ac_5517_4327_a6cc_4c7c105ac449.slice/crio-51c18754c18c356197081813e933a7dc65bd177c64a963a8ffedd67d52332352 WatchSource:0}: Error finding container 51c18754c18c356197081813e933a7dc65bd177c64a963a8ffedd67d52332352: Status 404 returned error can't find the container with id 51c18754c18c356197081813e933a7dc65bd177c64a963a8ffedd67d52332352 Jul 22 11:50:44.095426 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:50:44.095392 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5465cf73_cdb9_46e8_ad04_70550bc615a8.slice/crio-3fa819664439a885c462cf792ded163aa0af6c9655ef69a4e176a9f54495f27e WatchSource:0}: Error finding container 3fa819664439a885c462cf792ded163aa0af6c9655ef69a4e176a9f54495f27e: Status 404 returned error can't find the container with id 3fa819664439a885c462cf792ded163aa0af6c9655ef69a4e176a9f54495f27e Jul 22 11:50:44.264946 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:44.264897 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t4dxk" event={"ID":"5465cf73-cdb9-46e8-ad04-70550bc615a8","Type":"ContainerStarted","Data":"3fa819664439a885c462cf792ded163aa0af6c9655ef69a4e176a9f54495f27e"} Jul 22 11:50:44.266251 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:44.266222 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vk2qp" event={"ID":"5d43201c-5806-4536-8da3-8ad8fa286eac","Type":"ContainerStarted","Data":"e2566be96605fd88c2573a2be5fa5e78d29ae8f20580ade4a3f3027ef5acfb69"} Jul 22 11:50:44.268253 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:44.268225 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pz6s9" event={"ID":"ae05e4be-c308-457f-aaeb-c47ae8637746","Type":"ContainerStarted","Data":"a24f5bdf8acdd5c27c3e7406298d3d691a95372b55f1dd408bc4a81342747fe5"} Jul 22 11:50:44.269401 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:44.269380 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-78b62" event={"ID":"07c0d7ac-5517-4327-a6cc-4c7c105ac449","Type":"ContainerStarted","Data":"51c18754c18c356197081813e933a7dc65bd177c64a963a8ffedd67d52332352"} Jul 22 11:50:44.504991 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:44.504912 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:44.507559 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:44.507530 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls\") pod \"image-registry-59868b474b-zfdvd\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:44.782271 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:44.782191 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f7k7c\"" Jul 22 11:50:44.790444 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:44.790403 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:44.985656 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:44.985626 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59868b474b-zfdvd"] Jul 22 11:50:44.991791 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:50:44.991756 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87f5690_c6af_4536_a87f_de51ac6cec05.slice/crio-78ea07b453efb60c692b2b066f5cd791968d1491892680f3a6fe24a83b513cfc WatchSource:0}: Error finding container 78ea07b453efb60c692b2b066f5cd791968d1491892680f3a6fe24a83b513cfc: Status 404 returned error can't find the container with id 78ea07b453efb60c692b2b066f5cd791968d1491892680f3a6fe24a83b513cfc Jul 22 11:50:45.280102 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:45.280063 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vk2qp" event={"ID":"5d43201c-5806-4536-8da3-8ad8fa286eac","Type":"ContainerStarted","Data":"00c3c587af6d2d0e3e1fdbf3d113a6efcb8b6d54dbe8bf938478322ec099d22b"} Jul 22 11:50:45.280617 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:45.280555 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vk2qp" event={"ID":"5d43201c-5806-4536-8da3-8ad8fa286eac","Type":"ContainerStarted","Data":"725395b0fa35dc6e61d64cbe90ded1a212a84e42c4f8e61df769009e7476bb74"} Jul 22 11:50:45.282529 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:45.282501 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59868b474b-zfdvd" event={"ID":"c87f5690-c6af-4536-a87f-de51ac6cec05","Type":"ContainerStarted","Data":"9dbcc342d103bbea623fee247e9d3d4c129d8fd024805d32337f50ff89fdff65"} Jul 22 11:50:45.282636 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:45.282593 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59868b474b-zfdvd" event={"ID":"c87f5690-c6af-4536-a87f-de51ac6cec05","Type":"ContainerStarted","Data":"78ea07b453efb60c692b2b066f5cd791968d1491892680f3a6fe24a83b513cfc"} Jul 22 11:50:45.283226 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:45.283209 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:50:45.303361 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:45.302978 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vk2qp" podStartSLOduration=66.363321187 podStartE2EDuration="1m7.302929012s" totalImagesPullingTime="939.607825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:45.30269009 +0000 UTC m=+68.051901698" watchObservedRunningTime="2026-07-22 11:50:45.302929012 +0000 UTC m=+68.052140618" Jul 22 11:50:45.327050 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:45.326992 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59868b474b-zfdvd" podStartSLOduration=36.326970005 podStartE2EDuration="36.326970005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:45.32579372 +0000 UTC m=+68.075005516" watchObservedRunningTime="2026-07-22 11:50:45.326970005 +0000 UTC m=+68.076181608" Jul 22 11:50:45.455693 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:45.455654 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:45.525274 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:45.525237 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:46.942573 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:46.942511 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:47.891323 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:47.890938 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw"] Jul 22 11:50:47.998575 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:47.998543 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw" Jul 22 11:50:48.003135 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.002453 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 11:50:48.003135 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.002771 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-lwlrq\"" Jul 22 11:50:48.003135 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.002969 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 11:50:48.003135 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.003088 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c"] Jul 22 11:50:48.003433 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.003179 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 22 11:50:48.003606 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.003585 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 11:50:48.052359 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.052321 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c"] Jul 22 11:50:48.052359 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.052364 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw"] Jul 22 11:50:48.052579 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.052464 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" Jul 22 11:50:48.052579 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.052477 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78"] Jul 22 11:50:48.056902 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.056881 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 22 11:50:48.064198 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.064169 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:48.073765 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.073740 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78"] Jul 22 11:50:48.073886 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.073772 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59868b474b-zfdvd"] Jul 22 11:50:48.073936 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.073889 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.097937 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.097912 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 22 11:50:48.098353 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.098327 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 22 11:50:48.098493 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.098473 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 22 11:50:48.098679 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.098661 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 22 11:50:48.100445 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.100424 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fff7d8d76-k746k"] Jul 22 11:50:48.133692 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.133658 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjsc\" (UniqueName: \"kubernetes.io/projected/22646268-0da2-49bc-a6fc-c987446ad5b1-kube-api-access-9tjsc\") pod \"managed-serviceaccount-addon-agent-558f44bfb4-wvrfw\" (UID: \"22646268-0da2-49bc-a6fc-c987446ad5b1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw" Jul 22 11:50:48.133692 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.133695 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22646268-0da2-49bc-a6fc-c987446ad5b1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-558f44bfb4-wvrfw\" (UID: \"22646268-0da2-49bc-a6fc-c987446ad5b1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw" Jul 22 11:50:48.143669 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.143595 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.157640 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.157597 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fff7d8d76-k746k"] Jul 22 11:50:48.234902 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.234870 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/89074f62-c3b6-4e01-b74f-75a42bd274fe-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.234902 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.234905 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89074f62-c3b6-4e01-b74f-75a42bd274fe-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.235102 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.234927 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-bound-sa-token\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.235102 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.234951 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x6fv\" (UniqueName: \"kubernetes.io/projected/89074f62-c3b6-4e01-b74f-75a42bd274fe-kube-api-access-5x6fv\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.235102 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235000 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-image-registry-private-configuration\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.235102 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235044 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/78fefa72-8d40-41f6-9f95-33822dcb4f58-klusterlet-config\") pod \"klusterlet-addon-workmgr-b4fdc44cf-jq92c\" (UID: \"78fefa72-8d40-41f6-9f95-33822dcb4f58\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" Jul 22 11:50:48.235102 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235065 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxnln\" (UniqueName: \"kubernetes.io/projected/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-kube-api-access-jxnln\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.235102 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235089 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh4sf\" (UniqueName: \"kubernetes.io/projected/78fefa72-8d40-41f6-9f95-33822dcb4f58-kube-api-access-vh4sf\") pod \"klusterlet-addon-workmgr-b4fdc44cf-jq92c\" (UID: \"78fefa72-8d40-41f6-9f95-33822dcb4f58\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" Jul 22 11:50:48.235331 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235132 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-registry-tls\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.235331 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235165 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/89074f62-c3b6-4e01-b74f-75a42bd274fe-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.235331 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235214 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/89074f62-c3b6-4e01-b74f-75a42bd274fe-hub\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.235331 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235253 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-ca-trust-extracted\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.235331 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235289 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22646268-0da2-49bc-a6fc-c987446ad5b1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-558f44bfb4-wvrfw\" (UID: \"22646268-0da2-49bc-a6fc-c987446ad5b1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw" Jul 22 11:50:48.235331 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235311 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-registry-certificates\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.235548 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235358 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-trusted-ca\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.235548 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235398 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/89074f62-c3b6-4e01-b74f-75a42bd274fe-ca\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.235548 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235487 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78fefa72-8d40-41f6-9f95-33822dcb4f58-tmp\") pod \"klusterlet-addon-workmgr-b4fdc44cf-jq92c\" (UID: \"78fefa72-8d40-41f6-9f95-33822dcb4f58\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" Jul 22 11:50:48.235548 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235520 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-installation-pull-secrets\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.235721 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.235596 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjsc\" (UniqueName: \"kubernetes.io/projected/22646268-0da2-49bc-a6fc-c987446ad5b1-kube-api-access-9tjsc\") pod \"managed-serviceaccount-addon-agent-558f44bfb4-wvrfw\" (UID: \"22646268-0da2-49bc-a6fc-c987446ad5b1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw" Jul 22 11:50:48.238023 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.238006 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22646268-0da2-49bc-a6fc-c987446ad5b1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-558f44bfb4-wvrfw\" (UID: \"22646268-0da2-49bc-a6fc-c987446ad5b1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw" Jul 22 11:50:48.253849 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.253817 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjsc\" (UniqueName: \"kubernetes.io/projected/22646268-0da2-49bc-a6fc-c987446ad5b1-kube-api-access-9tjsc\") pod \"managed-serviceaccount-addon-agent-558f44bfb4-wvrfw\" (UID: \"22646268-0da2-49bc-a6fc-c987446ad5b1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw" Jul 22 11:50:48.314427 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.314393 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:48.318190 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.318164 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw" Jul 22 11:50:48.336289 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336248 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-trusted-ca\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.336289 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336293 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/89074f62-c3b6-4e01-b74f-75a42bd274fe-ca\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.336508 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336329 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78fefa72-8d40-41f6-9f95-33822dcb4f58-tmp\") pod \"klusterlet-addon-workmgr-b4fdc44cf-jq92c\" (UID: \"78fefa72-8d40-41f6-9f95-33822dcb4f58\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" Jul 22 11:50:48.336508 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336361 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-installation-pull-secrets\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.336508 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336411 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/89074f62-c3b6-4e01-b74f-75a42bd274fe-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.336508 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336436 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89074f62-c3b6-4e01-b74f-75a42bd274fe-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.336508 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336470 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-bound-sa-token\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.336739 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336511 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x6fv\" (UniqueName: \"kubernetes.io/projected/89074f62-c3b6-4e01-b74f-75a42bd274fe-kube-api-access-5x6fv\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.336739 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336543 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-image-registry-private-configuration\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.336739 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336580 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/78fefa72-8d40-41f6-9f95-33822dcb4f58-klusterlet-config\") pod \"klusterlet-addon-workmgr-b4fdc44cf-jq92c\" (UID: \"78fefa72-8d40-41f6-9f95-33822dcb4f58\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" Jul 22 11:50:48.336739 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336612 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxnln\" (UniqueName: \"kubernetes.io/projected/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-kube-api-access-jxnln\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.336739 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336649 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh4sf\" (UniqueName: \"kubernetes.io/projected/78fefa72-8d40-41f6-9f95-33822dcb4f58-kube-api-access-vh4sf\") pod \"klusterlet-addon-workmgr-b4fdc44cf-jq92c\" (UID: \"78fefa72-8d40-41f6-9f95-33822dcb4f58\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" Jul 22 11:50:48.336739 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336682 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-registry-tls\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.336739 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336719 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/89074f62-c3b6-4e01-b74f-75a42bd274fe-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.337056 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336755 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/89074f62-c3b6-4e01-b74f-75a42bd274fe-hub\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.337056 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336781 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-ca-trust-extracted\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.337056 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.336813 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-registry-certificates\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.337677 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.337654 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-registry-certificates\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.338671 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.338642 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-ca-trust-extracted\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.338810 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.338785 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-trusted-ca\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.339708 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.339681 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78fefa72-8d40-41f6-9f95-33822dcb4f58-tmp\") pod \"klusterlet-addon-workmgr-b4fdc44cf-jq92c\" (UID: \"78fefa72-8d40-41f6-9f95-33822dcb4f58\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" Jul 22 11:50:48.340357 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.339841 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/89074f62-c3b6-4e01-b74f-75a42bd274fe-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.341295 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.341268 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-image-registry-private-configuration\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.342145 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.342120 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/89074f62-c3b6-4e01-b74f-75a42bd274fe-ca\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.342260 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.342241 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-installation-pull-secrets\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.343030 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.342987 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-registry-tls\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.343158 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.343138 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/89074f62-c3b6-4e01-b74f-75a42bd274fe-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.343458 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.343437 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/89074f62-c3b6-4e01-b74f-75a42bd274fe-hub\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.343609 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.343593 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89074f62-c3b6-4e01-b74f-75a42bd274fe-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.344532 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.344508 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/78fefa72-8d40-41f6-9f95-33822dcb4f58-klusterlet-config\") pod \"klusterlet-addon-workmgr-b4fdc44cf-jq92c\" (UID: \"78fefa72-8d40-41f6-9f95-33822dcb4f58\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" Jul 22 11:50:48.362063 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.362035 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxnln\" (UniqueName: \"kubernetes.io/projected/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-kube-api-access-jxnln\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.362254 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.362230 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh4sf\" (UniqueName: \"kubernetes.io/projected/78fefa72-8d40-41f6-9f95-33822dcb4f58-kube-api-access-vh4sf\") pod \"klusterlet-addon-workmgr-b4fdc44cf-jq92c\" (UID: \"78fefa72-8d40-41f6-9f95-33822dcb4f58\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" Jul 22 11:50:48.362800 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.362783 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" Jul 22 11:50:48.366390 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.366370 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6c35d70-d0ce-43fb-acfd-37c85a4d2164-bound-sa-token\") pod \"image-registry-7fff7d8d76-k746k\" (UID: \"a6c35d70-d0ce-43fb-acfd-37c85a4d2164\") " pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.366902 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.366882 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x6fv\" (UniqueName: \"kubernetes.io/projected/89074f62-c3b6-4e01-b74f-75a42bd274fe-kube-api-access-5x6fv\") pod \"cluster-proxy-proxy-agent-7dbf4d44bf-mcc78\" (UID: \"89074f62-c3b6-4e01-b74f-75a42bd274fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.384760 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.384730 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" Jul 22 11:50:48.460669 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.460579 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:48.466385 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.466354 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:48.701557 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.701513 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:48.844830 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.844793 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:48.991929 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:48.991870 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:49.111387 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:49.111269 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:49.357772 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:49.357662 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:49.451750 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:49.451674 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:49.835184 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:49.835152 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:49.915445 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:49.915406 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:50.911142 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:50.911071 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c"] Jul 22 11:50:50.914818 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:50:50.914788 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78fefa72_8d40_41f6_9f95_33822dcb4f58.slice/crio-e22ac20d0dd96f0d1581c0399a150b9a4d2b6e15651fba01c518c1a0c39abf72 WatchSource:0}: Error finding container e22ac20d0dd96f0d1581c0399a150b9a4d2b6e15651fba01c518c1a0c39abf72: Status 404 returned error can't find the container with id e22ac20d0dd96f0d1581c0399a150b9a4d2b6e15651fba01c518c1a0c39abf72 Jul 22 11:50:50.974974 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:50.974884 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:51.124266 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.124235 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78"] Jul 22 11:50:51.127194 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:50:51.127168 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89074f62_c3b6_4e01_b74f_75a42bd274fe.slice/crio-2061509808cc7cda93b11498b336fb0306b748648e7291085a95145b7cd9012d WatchSource:0}: Error finding container 2061509808cc7cda93b11498b336fb0306b748648e7291085a95145b7cd9012d: Status 404 returned error can't find the container with id 2061509808cc7cda93b11498b336fb0306b748648e7291085a95145b7cd9012d Jul 22 11:50:51.133974 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:50:51.133943 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22646268_0da2_49bc_a6fc_c987446ad5b1.slice/crio-96c93e1de833388d2f40a3cd291a33026d009b381289ff0f6edaebcb624cba49 WatchSource:0}: Error finding container 96c93e1de833388d2f40a3cd291a33026d009b381289ff0f6edaebcb624cba49: Status 404 returned error can't find the container with id 96c93e1de833388d2f40a3cd291a33026d009b381289ff0f6edaebcb624cba49 Jul 22 11:50:51.134094 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.133989 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw"] Jul 22 11:50:51.134839 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:50:51.134807 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c35d70_d0ce_43fb_acfd_37c85a4d2164.slice/crio-c30f5dfe875164ea3caef6e9ff4941e392fbc8b6c1f810fc615cc36f63df5bf0 WatchSource:0}: Error finding container c30f5dfe875164ea3caef6e9ff4941e392fbc8b6c1f810fc615cc36f63df5bf0: Status 404 returned error can't find the container with id c30f5dfe875164ea3caef6e9ff4941e392fbc8b6c1f810fc615cc36f63df5bf0 Jul 22 11:50:51.134949 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.134928 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fff7d8d76-k746k"] Jul 22 11:50:51.148464 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.148433 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:51.305544 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.305460 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw" event={"ID":"22646268-0da2-49bc-a6fc-c987446ad5b1","Type":"ContainerStarted","Data":"96c93e1de833388d2f40a3cd291a33026d009b381289ff0f6edaebcb624cba49"} Jul 22 11:50:51.307549 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.307527 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pz6s9" event={"ID":"ae05e4be-c308-457f-aaeb-c47ae8637746","Type":"ContainerStarted","Data":"17f2cdca4830d0f77308b089cb13b41772bca3ab1145982f7963f72da0b09c9c"} Jul 22 11:50:51.309051 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.309030 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-78b62" event={"ID":"07c0d7ac-5517-4327-a6cc-4c7c105ac449","Type":"ContainerStarted","Data":"56d7d0a427ab757b8419fe6dfbcf5656cc853ae8035118e0ef7389e7f2a5f29f"} Jul 22 11:50:51.309180 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.309076 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:50:51.310626 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.310606 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" event={"ID":"a6c35d70-d0ce-43fb-acfd-37c85a4d2164","Type":"ContainerStarted","Data":"6305e07344ae1caf5c305985daf413860d84992918ba945245b10af04c747ab9"} Jul 22 11:50:51.310732 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.310643 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" event={"ID":"a6c35d70-d0ce-43fb-acfd-37c85a4d2164","Type":"ContainerStarted","Data":"c30f5dfe875164ea3caef6e9ff4941e392fbc8b6c1f810fc615cc36f63df5bf0"} Jul 22 11:50:51.310732 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.310669 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:50:51.311854 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.311828 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" event={"ID":"78fefa72-8d40-41f6-9f95-33822dcb4f58","Type":"ContainerStarted","Data":"e22ac20d0dd96f0d1581c0399a150b9a4d2b6e15651fba01c518c1a0c39abf72"} Jul 22 11:50:51.313294 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.313270 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t4dxk" event={"ID":"5465cf73-cdb9-46e8-ad04-70550bc615a8","Type":"ContainerStarted","Data":"822144c4178bb53dc7f08b7c926f53130499f19dd86e44ab574ce295dbb670f2"} Jul 22 11:50:51.314466 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.314445 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" event={"ID":"89074f62-c3b6-4e01-b74f-75a42bd274fe","Type":"ContainerStarted","Data":"2061509808cc7cda93b11498b336fb0306b748648e7291085a95145b7cd9012d"} Jul 22 11:50:51.333341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.333287 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pz6s9" podStartSLOduration=17.839261951 podStartE2EDuration="33.333270814s" totalImagesPullingTime="15.494008863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:50:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:51.332097275 +0000 UTC m=+74.081308876" watchObservedRunningTime="2026-07-22 11:50:51.333270814 +0000 UTC m=+74.082482415" Jul 22 11:50:51.361300 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.361246 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" podStartSLOduration=3.361232145 podStartE2EDuration="3.361232145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:51.360307239 +0000 UTC m=+74.109518842" watchObservedRunningTime="2026-07-22 11:50:51.361232145 +0000 UTC m=+74.110443747" Jul 22 11:50:51.378850 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.378795 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t4dxk" podStartSLOduration=66.759956227 podStartE2EDuration="1m13.378778859s" totalImagesPullingTime="6.618822632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:51.377629172 +0000 UTC m=+74.126840773" watchObservedRunningTime="2026-07-22 11:50:51.378778859 +0000 UTC m=+74.127990439" Jul 22 11:50:51.402082 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.402029 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-78b62" podStartSLOduration=66.791466236 podStartE2EDuration="1m13.402011918s" totalImagesPullingTime="6.610545682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:51.396809849 +0000 UTC m=+74.146021453" watchObservedRunningTime="2026-07-22 11:50:51.402011918 +0000 UTC m=+74.151223520" Jul 22 11:50:51.696785 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.696750 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:51.790287 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.790224 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:51.892620 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.892566 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:51.981605 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:51.981527 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:54.901978 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:54.901935 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:55.024412 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:55.024349 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:55.972126 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:55.972080 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:56.073514 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:56.073481 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:56.173490 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:56.173455 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:57.341393 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:57.341358 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" event={"ID":"78fefa72-8d40-41f6-9f95-33822dcb4f58","Type":"ContainerStarted","Data":"286091bdfa75bdf996077b721a8648eb4c2a97e5ba7612dd18f0b65e3493b8de"} Jul 22 11:50:57.341866 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:57.341537 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" Jul 22 11:50:57.343406 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:57.343382 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" Jul 22 11:50:57.343815 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:57.343791 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" event={"ID":"89074f62-c3b6-4e01-b74f-75a42bd274fe","Type":"ContainerStarted","Data":"667a316f2f3b25b5f3626d22e724b61afad38bf51b72e5436d816daffb5e6b7a"} Jul 22 11:50:57.343895 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:57.343826 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" event={"ID":"89074f62-c3b6-4e01-b74f-75a42bd274fe","Type":"ContainerStarted","Data":"70780f06dd444040e791896a55569b10d9794cfdd664d02839eca65772edd2d2"} Jul 22 11:50:57.343895 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:57.343839 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" event={"ID":"89074f62-c3b6-4e01-b74f-75a42bd274fe","Type":"ContainerStarted","Data":"8d613a05d00c220ce5d31b0e9d730871f10c0be1542fab4b1d0ebfe7fe2c5eaf"} Jul 22 11:50:57.345327 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:57.345305 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw" event={"ID":"22646268-0da2-49bc-a6fc-c987446ad5b1","Type":"ContainerStarted","Data":"b88d4757b0aca99297ed4d23c2a9c8b74e45fa0aac1c8f6594cd61652e1aecee"} Jul 22 11:50:57.361060 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:57.361007 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4fdc44cf-jq92c" podStartSLOduration=4.733291494 podStartE2EDuration="10.360993289s" totalImagesPullingTime="5.627701795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:57.360372575 +0000 UTC m=+80.109584192" watchObservedRunningTime="2026-07-22 11:50:57.360993289 +0000 UTC m=+80.110204888" Jul 22 11:50:57.381822 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:57.381771 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf4d44bf-mcc78" podStartSLOduration=4.964857677 podStartE2EDuration="10.38175651s" totalImagesPullingTime="5.416898833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:57.380586819 +0000 UTC m=+80.129798421" watchObservedRunningTime="2026-07-22 11:50:57.38175651 +0000 UTC m=+80.130968112" Jul 22 11:50:57.397711 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:57.397671 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558f44bfb4-wvrfw" podStartSLOduration=5.005289439 podStartE2EDuration="10.39765664s" totalImagesPullingTime="5.392367201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:50:57.396976119 +0000 UTC m=+80.146187722" watchObservedRunningTime="2026-07-22 11:50:57.39765664 +0000 UTC m=+80.146868243" Jul 22 11:50:57.914436 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:57.914396 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:58.015625 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:58.015586 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:58.875076 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:58.875044 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:50:58.979744 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:50:58.979703 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:00.900820 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:00.900787 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:01.005687 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:01.005650 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:01.341942 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:01.341906 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:02.952080 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:02.952049 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:03.053524 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:03.053492 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:03.995249 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:03.995218 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:05.182097 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.182056 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:05.349378 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.349315 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:05.501249 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.501156 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:05.516400 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.516371 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vtpbp"] Jul 22 11:51:05.522603 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.522585 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.525125 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:51:05.525081 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-130-190.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" Jul 22 11:51:05.527834 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:51:05.527810 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"node-exporter-accelerators-collector-config\" is forbidden: User \"system:node:ip-10-0-130-190.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" Jul 22 11:51:05.528168 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:51:05.528134 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"metrics-client-ca\" is forbidden: User \"system:node:ip-10-0-130-190.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" Jul 22 11:51:05.528168 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:51:05.528155 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-130-190.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" Jul 22 11:51:05.528944 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.528924 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 11:51:05.529262 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.529243 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p2zfp\"" Jul 22 11:51:05.529410 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.529382 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 11:51:05.576048 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.576006 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-textfile\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.576340 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.576312 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c32bd875-338b-4c02-93ac-b1b1bb5c96da-sys\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.576463 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.576365 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.576463 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.576414 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-tls\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.576463 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.576444 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.576625 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.576480 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c32bd875-338b-4c02-93ac-b1b1bb5c96da-metrics-client-ca\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.576625 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.576508 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmwtd\" (UniqueName: \"kubernetes.io/projected/c32bd875-338b-4c02-93ac-b1b1bb5c96da-kube-api-access-pmwtd\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.576625 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.576532 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-wtmp\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.576625 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.576559 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c32bd875-338b-4c02-93ac-b1b1bb5c96da-root\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.667316 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.667278 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:05.677853 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.677824 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-tls\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.678025 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.677859 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.678025 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.677882 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c32bd875-338b-4c02-93ac-b1b1bb5c96da-metrics-client-ca\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.678025 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.677900 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwtd\" (UniqueName: \"kubernetes.io/projected/c32bd875-338b-4c02-93ac-b1b1bb5c96da-kube-api-access-pmwtd\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.678025 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.677915 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-wtmp\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.678025 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.678016 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c32bd875-338b-4c02-93ac-b1b1bb5c96da-root\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.678292 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.678065 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-textfile\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.678292 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.678101 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c32bd875-338b-4c02-93ac-b1b1bb5c96da-sys\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.678292 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.678152 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c32bd875-338b-4c02-93ac-b1b1bb5c96da-root\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.678292 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.678152 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-wtmp\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.678292 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.678194 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c32bd875-338b-4c02-93ac-b1b1bb5c96da-sys\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.678292 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.678154 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.678483 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.678420 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-textfile\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.692060 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.692029 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmwtd\" (UniqueName: \"kubernetes.io/projected/c32bd875-338b-4c02-93ac-b1b1bb5c96da-kube-api-access-pmwtd\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:05.841120 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.841070 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:05.931250 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:05.931195 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:06.078136 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:06.078082 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:06.171329 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:06.171227 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:06.390099 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:06.390065 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:06.512200 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:06.512082 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:06.652986 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:06.652930 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:06.663594 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:06.663570 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 11:51:06.669079 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:06.669061 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c32bd875-338b-4c02-93ac-b1b1bb5c96da-metrics-client-ca\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:06.678335 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:51:06.678317 2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-kube-rbac-proxy-config: failed to sync secret cache: timed out waiting for the condition Jul 22 11:51:06.678335 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:51:06.678326 2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: failed to sync secret cache: timed out waiting for the condition Jul 22 11:51:06.678432 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:51:06.678332 2555 configmap.go:193] Couldn't get configMap openshift-monitoring/node-exporter-accelerators-collector-config: failed to sync configmap cache: timed out waiting for the condition Jul 22 11:51:06.678432 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:51:06.678381 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-kube-rbac-proxy-config podName:c32bd875-338b-4c02-93ac-b1b1bb5c96da nodeName:}" failed. No retries permitted until 2026-07-22 11:51:07.178365479 +0000 UTC m=+89.927577061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-kube-rbac-proxy-config" (UniqueName: "kubernetes.io/secret/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-kube-rbac-proxy-config") pod "node-exporter-vtpbp" (UID: "c32bd875-338b-4c02-93ac-b1b1bb5c96da") : failed to sync secret cache: timed out waiting for the condition Jul 22 11:51:06.678432 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:51:06.678424 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-tls podName:c32bd875-338b-4c02-93ac-b1b1bb5c96da nodeName:}" failed. No retries permitted until 2026-07-22 11:51:07.178406571 +0000 UTC m=+89.927618164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-tls") pod "node-exporter-vtpbp" (UID: "c32bd875-338b-4c02-93ac-b1b1bb5c96da") : failed to sync secret cache: timed out waiting for the condition Jul 22 11:51:06.678538 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:51:06.678438 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-accelerators-collector-config podName:c32bd875-338b-4c02-93ac-b1b1bb5c96da nodeName:}" failed. No retries permitted until 2026-07-22 11:51:07.178430888 +0000 UTC m=+89.927642469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-accelerators-collector-config" (UniqueName: "kubernetes.io/configmap/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-accelerators-collector-config") pod "node-exporter-vtpbp" (UID: "c32bd875-338b-4c02-93ac-b1b1bb5c96da") : failed to sync configmap cache: timed out waiting for the condition Jul 22 11:51:06.815252 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:06.815171 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 11:51:06.928846 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:06.928811 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 11:51:06.938684 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:06.938664 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 11:51:07.190613 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:07.190581 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-tls\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:07.190613 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:07.190616 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:07.190883 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:07.190764 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:07.191092 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:07.191072 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:07.192875 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:07.192860 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-tls\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:07.192976 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:07.192959 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c32bd875-338b-4c02-93ac-b1b1bb5c96da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtpbp\" (UID: \"c32bd875-338b-4c02-93ac-b1b1bb5c96da\") " pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:07.332947 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:07.332916 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vtpbp" Jul 22 11:51:07.345205 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:51:07.345177 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32bd875_338b_4c02_93ac_b1b1bb5c96da.slice/crio-d6f7f296eb3ae96143c4c1c09e55b8e03f40e62ac15e4f740b6a6cfa321a6918 WatchSource:0}: Error finding container d6f7f296eb3ae96143c4c1c09e55b8e03f40e62ac15e4f740b6a6cfa321a6918: Status 404 returned error can't find the container with id d6f7f296eb3ae96143c4c1c09e55b8e03f40e62ac15e4f740b6a6cfa321a6918 Jul 22 11:51:07.383847 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:07.383815 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtpbp" event={"ID":"c32bd875-338b-4c02-93ac-b1b1bb5c96da","Type":"ContainerStarted","Data":"d6f7f296eb3ae96143c4c1c09e55b8e03f40e62ac15e4f740b6a6cfa321a6918"} Jul 22 11:51:08.301072 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:08.301043 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:51:08.389447 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:08.389358 2555 generic.go:356] "Generic (PLEG): container finished" podID="c32bd875-338b-4c02-93ac-b1b1bb5c96da" containerID="6824a1539fde1dfef08464930698c79fccd1a9262754a7a1d70e32f4577a7ccc" exitCode=0 Jul 22 11:51:08.389447 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:08.389397 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtpbp" event={"ID":"c32bd875-338b-4c02-93ac-b1b1bb5c96da","Type":"ContainerDied","Data":"6824a1539fde1dfef08464930698c79fccd1a9262754a7a1d70e32f4577a7ccc"} Jul 22 11:51:08.490644 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:08.490606 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:08.587884 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:08.587829 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:08.696293 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:08.696206 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:08.813612 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:08.813583 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:08.916716 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:08.916675 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:09.019424 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:09.019336 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:09.395095 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:09.395062 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtpbp" event={"ID":"c32bd875-338b-4c02-93ac-b1b1bb5c96da","Type":"ContainerStarted","Data":"48cf6a6788ba47751d0b53c4f9571d8e4a3a0200960e51a35537827ee53c8993"} Jul 22 11:51:09.395491 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:09.395134 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtpbp" event={"ID":"c32bd875-338b-4c02-93ac-b1b1bb5c96da","Type":"ContainerStarted","Data":"9ff3274786b073f4687c6683a10dba8b1a06c31675e7a93eec34950c20fe9c7d"} Jul 22 11:51:09.419630 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:09.419559 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vtpbp" podStartSLOduration=3.571799221 podStartE2EDuration="4.419539445s" totalImagesPullingTime="783.095679ms" totalInitContainerRuntime="64.644545ms" isStatefulPod=true podCreationTimestamp="2026-07-22 11:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:51:09.419162 +0000 UTC m=+92.168373603" watchObservedRunningTime="2026-07-22 11:51:09.419539445 +0000 UTC m=+92.168751052" Jul 22 11:51:10.224313 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:10.224273 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:10.334993 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:10.334957 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:10.426238 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:10.426200 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:10.652022 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:10.651985 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:10.734042 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:10.733996 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:11.872160 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:11.872074 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:12.325148 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:12.325055 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fff7d8d76-k746k" Jul 22 11:51:12.607346 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:12.607308 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:12.708193 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:12.708156 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:13.269121 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:13.269075 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:13.315783 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:13.315745 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59868b474b-zfdvd" podUID="c87f5690-c6af-4536-a87f-de51ac6cec05" containerName="registry" containerID="cri-o://9dbcc342d103bbea623fee247e9d3d4c129d8fd024805d32337f50ff89fdff65" gracePeriod=30 Jul 22 11:51:13.385841 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:13.385796 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:14.091072 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.091044 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:51:14.144822 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.144795 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-certificates\") pod \"c87f5690-c6af-4536-a87f-de51ac6cec05\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " Jul 22 11:51:14.144822 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.144828 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-bound-sa-token\") pod \"c87f5690-c6af-4536-a87f-de51ac6cec05\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " Jul 22 11:51:14.145055 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.144863 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnzf6\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-kube-api-access-lnzf6\") pod \"c87f5690-c6af-4536-a87f-de51ac6cec05\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " Jul 22 11:51:14.145055 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.144899 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls\") pod \"c87f5690-c6af-4536-a87f-de51ac6cec05\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " Jul 22 11:51:14.145055 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.144919 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c87f5690-c6af-4536-a87f-de51ac6cec05-installation-pull-secrets\") pod \"c87f5690-c6af-4536-a87f-de51ac6cec05\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " Jul 22 11:51:14.145055 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.144945 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c87f5690-c6af-4536-a87f-de51ac6cec05-image-registry-private-configuration\") pod \"c87f5690-c6af-4536-a87f-de51ac6cec05\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " Jul 22 11:51:14.145055 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.144985 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c87f5690-c6af-4536-a87f-de51ac6cec05-trusted-ca\") pod \"c87f5690-c6af-4536-a87f-de51ac6cec05\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " Jul 22 11:51:14.145055 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.145002 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c87f5690-c6af-4536-a87f-de51ac6cec05-ca-trust-extracted\") pod \"c87f5690-c6af-4536-a87f-de51ac6cec05\" (UID: \"c87f5690-c6af-4536-a87f-de51ac6cec05\") " Jul 22 11:51:14.145406 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.145375 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c87f5690-c6af-4536-a87f-de51ac6cec05" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:51:14.145590 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.145540 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87f5690-c6af-4536-a87f-de51ac6cec05-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c87f5690-c6af-4536-a87f-de51ac6cec05" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:51:14.147423 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.147384 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c87f5690-c6af-4536-a87f-de51ac6cec05" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:51:14.147589 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.147568 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c87f5690-c6af-4536-a87f-de51ac6cec05-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c87f5690-c6af-4536-a87f-de51ac6cec05" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:51:14.147669 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.147642 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c87f5690-c6af-4536-a87f-de51ac6cec05-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c87f5690-c6af-4536-a87f-de51ac6cec05" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:51:14.147814 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.147796 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c87f5690-c6af-4536-a87f-de51ac6cec05" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:51:14.147996 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.147927 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-kube-api-access-lnzf6" (OuterVolumeSpecName: "kube-api-access-lnzf6") pod "c87f5690-c6af-4536-a87f-de51ac6cec05" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05"). InnerVolumeSpecName "kube-api-access-lnzf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:51:14.153451 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.153429 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87f5690-c6af-4536-a87f-de51ac6cec05-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c87f5690-c6af-4536-a87f-de51ac6cec05" (UID: "c87f5690-c6af-4536-a87f-de51ac6cec05"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:51:14.246309 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.246233 2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-bound-sa-token\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 11:51:14.246309 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.246260 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnzf6\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-kube-api-access-lnzf6\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 11:51:14.246309 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.246271 2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-tls\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 11:51:14.246309 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.246284 2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c87f5690-c6af-4536-a87f-de51ac6cec05-installation-pull-secrets\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 11:51:14.246309 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.246293 2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c87f5690-c6af-4536-a87f-de51ac6cec05-image-registry-private-configuration\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 11:51:14.246309 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.246302 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c87f5690-c6af-4536-a87f-de51ac6cec05-trusted-ca\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 11:51:14.246309 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.246313 2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c87f5690-c6af-4536-a87f-de51ac6cec05-ca-trust-extracted\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 11:51:14.246629 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.246321 2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c87f5690-c6af-4536-a87f-de51ac6cec05-registry-certificates\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 11:51:14.414271 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.414241 2555 generic.go:356] "Generic (PLEG): container finished" podID="c87f5690-c6af-4536-a87f-de51ac6cec05" containerID="9dbcc342d103bbea623fee247e9d3d4c129d8fd024805d32337f50ff89fdff65" exitCode=0 Jul 22 11:51:14.414734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.414284 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59868b474b-zfdvd" Jul 22 11:51:14.414734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.414313 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59868b474b-zfdvd" event={"ID":"c87f5690-c6af-4536-a87f-de51ac6cec05","Type":"ContainerDied","Data":"9dbcc342d103bbea623fee247e9d3d4c129d8fd024805d32337f50ff89fdff65"} Jul 22 11:51:14.414734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.414351 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59868b474b-zfdvd" event={"ID":"c87f5690-c6af-4536-a87f-de51ac6cec05","Type":"ContainerDied","Data":"78ea07b453efb60c692b2b066f5cd791968d1491892680f3a6fe24a83b513cfc"} Jul 22 11:51:14.414734 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.414370 2555 scope.go:117] "RemoveContainer" containerID="9dbcc342d103bbea623fee247e9d3d4c129d8fd024805d32337f50ff89fdff65" Jul 22 11:51:14.423362 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.423344 2555 scope.go:117] "RemoveContainer" containerID="9dbcc342d103bbea623fee247e9d3d4c129d8fd024805d32337f50ff89fdff65" Jul 22 11:51:14.423664 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:51:14.423646 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbcc342d103bbea623fee247e9d3d4c129d8fd024805d32337f50ff89fdff65\": container with ID starting with 9dbcc342d103bbea623fee247e9d3d4c129d8fd024805d32337f50ff89fdff65 not found: ID does not exist" containerID="9dbcc342d103bbea623fee247e9d3d4c129d8fd024805d32337f50ff89fdff65" Jul 22 11:51:14.423711 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.423674 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbcc342d103bbea623fee247e9d3d4c129d8fd024805d32337f50ff89fdff65"} err="failed to get container status \"9dbcc342d103bbea623fee247e9d3d4c129d8fd024805d32337f50ff89fdff65\": rpc error: code = NotFound desc = could not find container \"9dbcc342d103bbea623fee247e9d3d4c129d8fd024805d32337f50ff89fdff65\": container with ID starting with 9dbcc342d103bbea623fee247e9d3d4c129d8fd024805d32337f50ff89fdff65 not found: ID does not exist" Jul 22 11:51:14.438329 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.438305 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59868b474b-zfdvd"] Jul 22 11:51:14.441403 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:14.441380 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59868b474b-zfdvd"] Jul 22 11:51:15.957896 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:15.957850 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c87f5690-c6af-4536-a87f-de51ac6cec05" path="/var/lib/kubelet/pods/c87f5690-c6af-4536-a87f-de51ac6cec05/volumes" Jul 22 11:51:17.727349 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:17.727312 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:17.826341 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:17.826270 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:17.918864 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:17.918830 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:21.212931 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:21.212894 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:21.323506 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:21.323465 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:22.322862 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:22.322833 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-78b62" Jul 22 11:51:25.636755 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:25.636721 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:25.735934 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:25.735894 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:35.787925 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:35.787892 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:35.865191 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:35.865153 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:40.635259 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:40.635224 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:40.713962 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:51:40.713932 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:24.843234 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:24.843189 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:24.944373 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:24.944334 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:34.203401 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:34.203305 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:34.897637 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:34.897591 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:34.992305 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:34.992264 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:35.091747 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:35.091711 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:35.192004 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:35.191894 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:35.622921 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:35.622871 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:35.725223 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:35.725183 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:40.072375 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:40.072342 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:40.172913 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:40.172872 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:52.521131 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:52.521078 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:52.579938 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:52.579885 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:53.343677 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:53.343639 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:53.436797 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:53.436757 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:53.508064 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:53.508033 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:54.986350 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:54.986305 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:55.657041 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:55.656995 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:55.758062 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:52:55.758028 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:25.030258 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:25.030220 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:25.130191 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:25.130158 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:26.847218 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:26.847179 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:45.888591 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:45.888562 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:45.975866 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:45.975800 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:46.757653 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:46.757617 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:48.064378 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:48.064343 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:50.838420 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:50.838387 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:51.070873 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:51.070829 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:53.157456 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:53.157417 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:53.585238 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:53.585207 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:55.743555 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:55.743516 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:55.957205 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:55.957172 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:57.723517 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:57.723487 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:57.939200 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:53:57.939164 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:03.028446 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:03.028353 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:05.087593 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:05.087562 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:05.233089 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:05.233037 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:10.287180 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:10.287143 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:10.388090 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:10.388058 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:12.041956 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:12.041920 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:22.882795 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:22.882760 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:23.072255 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:23.072217 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:24.167924 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:24.167887 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:24.276581 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:24.276546 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:24.842917 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:24.842880 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:25.122820 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.122708 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:25.263694 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.263666 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:25.270568 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.270544 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv"] Jul 22 11:54:25.270772 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.270761 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c87f5690-c6af-4536-a87f-de51ac6cec05" containerName="registry" Jul 22 11:54:25.270812 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.270773 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c87f5690-c6af-4536-a87f-de51ac6cec05" containerName="registry" Jul 22 11:54:25.270845 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.270825 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c87f5690-c6af-4536-a87f-de51ac6cec05" containerName="registry" Jul 22 11:54:25.273572 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.273558 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.277733 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.277616 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 11:54:25.278838 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.278803 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 11:54:25.279149 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.279135 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 11:54:25.279656 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.279633 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 11:54:25.287653 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.287635 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x6j7c\"" Jul 22 11:54:25.290537 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.290517 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6476de55-5def-4b98-a148-8bfc54acf4d8-manifests\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.290611 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.290546 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6476de55-5def-4b98-a148-8bfc54acf4d8-tmp\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.290611 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.290571 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6476de55-5def-4b98-a148-8bfc54acf4d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.290679 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.290641 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4tks\" (UniqueName: \"kubernetes.io/projected/6476de55-5def-4b98-a148-8bfc54acf4d8-kube-api-access-v4tks\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.290679 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.290669 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6476de55-5def-4b98-a148-8bfc54acf4d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.306886 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.306861 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv"] Jul 22 11:54:25.391227 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.391141 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6476de55-5def-4b98-a148-8bfc54acf4d8-manifests\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.391227 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.391182 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6476de55-5def-4b98-a148-8bfc54acf4d8-tmp\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.391227 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.391202 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6476de55-5def-4b98-a148-8bfc54acf4d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.391500 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.391253 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4tks\" (UniqueName: \"kubernetes.io/projected/6476de55-5def-4b98-a148-8bfc54acf4d8-kube-api-access-v4tks\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.391500 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.391278 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6476de55-5def-4b98-a148-8bfc54acf4d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.391604 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.391511 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6476de55-5def-4b98-a148-8bfc54acf4d8-manifests\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.393663 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.393634 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6476de55-5def-4b98-a148-8bfc54acf4d8-tmp\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.393768 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.393695 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6476de55-5def-4b98-a148-8bfc54acf4d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.393768 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.393727 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6476de55-5def-4b98-a148-8bfc54acf4d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.403253 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.403231 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4tks\" (UniqueName: \"kubernetes.io/projected/6476de55-5def-4b98-a148-8bfc54acf4d8-kube-api-access-v4tks\") pod \"opendatahub-operator-controller-manager-c4d6d56dd-dbzrv\" (UID: \"6476de55-5def-4b98-a148-8bfc54acf4d8\") " pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.491588 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.491548 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:25.583256 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.583224 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:25.627078 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.627048 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:25.725641 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.725612 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv"] Jul 22 11:54:25.727681 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:54:25.727655 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6476de55_5def_4b98_a148_8bfc54acf4d8.slice/crio-0e5ce52cc25450e679a43e50b1e026087598a624e219d26094202f051a7982dc WatchSource:0}: Error finding container 0e5ce52cc25450e679a43e50b1e026087598a624e219d26094202f051a7982dc: Status 404 returned error can't find the container with id 0e5ce52cc25450e679a43e50b1e026087598a624e219d26094202f051a7982dc Jul 22 11:54:25.753156 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.753123 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:25.870593 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.870554 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:25.977053 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:25.976968 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:26.043895 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:26.043858 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" event={"ID":"6476de55-5def-4b98-a148-8bfc54acf4d8","Type":"ContainerStarted","Data":"0e5ce52cc25450e679a43e50b1e026087598a624e219d26094202f051a7982dc"} Jul 22 11:54:29.058359 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:29.058275 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" event={"ID":"6476de55-5def-4b98-a148-8bfc54acf4d8","Type":"ContainerStarted","Data":"62506495d6b11d94c4970cbb1659cec28d6afaed5d19e8ac219e79fc89fd567c"} Jul 22 11:54:30.062670 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:30.062582 2555 generic.go:356] "Generic (PLEG): container finished" podID="6476de55-5def-4b98-a148-8bfc54acf4d8" containerID="62506495d6b11d94c4970cbb1659cec28d6afaed5d19e8ac219e79fc89fd567c" exitCode=0 Jul 22 11:54:30.063071 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:30.062668 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" event={"ID":"6476de55-5def-4b98-a148-8bfc54acf4d8","Type":"ContainerDied","Data":"62506495d6b11d94c4970cbb1659cec28d6afaed5d19e8ac219e79fc89fd567c"} Jul 22 11:54:31.068522 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:31.068425 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" event={"ID":"6476de55-5def-4b98-a148-8bfc54acf4d8","Type":"ContainerStarted","Data":"78280df42e1137768fafa78f65ce4c2da0e648e6517cb84ee4f31c9e16918b51"} Jul 22 11:54:31.068907 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:31.068552 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:31.095899 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:31.095847 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" podStartSLOduration=1.578313421 podStartE2EDuration="6.09583163s" totalImagesPullingTime="3.487720621s" totalInitContainerRuntime="1.029797588s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:54:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:54:31.093903888 +0000 UTC m=+293.843115490" watchObservedRunningTime="2026-07-22 11:54:31.09583163 +0000 UTC m=+293.845043231" Jul 22 11:54:37.618923 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:37.618883 2555 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 11:54:42.009197 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:42.009166 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:42.074161 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:42.074131 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c4d6d56dd-dbzrv" Jul 22 11:54:43.565426 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:43.565390 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:46.261587 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:46.261546 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:46.482130 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:46.482073 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:46.829050 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:46.829015 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-29bk6"] Jul 22 11:54:46.831599 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:46.831573 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" Jul 22 11:54:46.838367 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:46.838347 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-665vk\"" Jul 22 11:54:46.838837 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:46.838819 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 22 11:54:46.845663 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:46.845633 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-29bk6"] Jul 22 11:54:46.857051 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:46.857007 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db6bec41-3b3b-4cff-ae06-9091be5196fc-cert\") pod \"odh-model-controller-7998c67dc4-29bk6\" (UID: \"db6bec41-3b3b-4cff-ae06-9091be5196fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" Jul 22 11:54:46.857251 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:46.857072 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mnsp\" (UniqueName: \"kubernetes.io/projected/db6bec41-3b3b-4cff-ae06-9091be5196fc-kube-api-access-9mnsp\") pod \"odh-model-controller-7998c67dc4-29bk6\" (UID: \"db6bec41-3b3b-4cff-ae06-9091be5196fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" Jul 22 11:54:46.957575 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:46.957529 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db6bec41-3b3b-4cff-ae06-9091be5196fc-cert\") pod \"odh-model-controller-7998c67dc4-29bk6\" (UID: \"db6bec41-3b3b-4cff-ae06-9091be5196fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" Jul 22 11:54:46.957776 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:46.957591 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mnsp\" (UniqueName: \"kubernetes.io/projected/db6bec41-3b3b-4cff-ae06-9091be5196fc-kube-api-access-9mnsp\") pod \"odh-model-controller-7998c67dc4-29bk6\" (UID: \"db6bec41-3b3b-4cff-ae06-9091be5196fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" Jul 22 11:54:46.957776 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:54:46.957726 2555 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 22 11:54:46.957896 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:54:46.957804 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db6bec41-3b3b-4cff-ae06-9091be5196fc-cert podName:db6bec41-3b3b-4cff-ae06-9091be5196fc nodeName:}" failed. No retries permitted until 2026-07-22 11:54:47.457781269 +0000 UTC m=+310.206992868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/db6bec41-3b3b-4cff-ae06-9091be5196fc-cert") pod "odh-model-controller-7998c67dc4-29bk6" (UID: "db6bec41-3b3b-4cff-ae06-9091be5196fc") : secret "odh-model-controller-webhook-cert" not found Jul 22 11:54:46.958690 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:46.958655 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:46.987516 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:46.987403 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mnsp\" (UniqueName: \"kubernetes.io/projected/db6bec41-3b3b-4cff-ae06-9091be5196fc-kube-api-access-9mnsp\") pod \"odh-model-controller-7998c67dc4-29bk6\" (UID: \"db6bec41-3b3b-4cff-ae06-9091be5196fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" Jul 22 11:54:47.166729 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:47.166694 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:47.461749 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:47.461661 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db6bec41-3b3b-4cff-ae06-9091be5196fc-cert\") pod \"odh-model-controller-7998c67dc4-29bk6\" (UID: \"db6bec41-3b3b-4cff-ae06-9091be5196fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" Jul 22 11:54:47.464206 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:47.464182 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db6bec41-3b3b-4cff-ae06-9091be5196fc-cert\") pod \"odh-model-controller-7998c67dc4-29bk6\" (UID: \"db6bec41-3b3b-4cff-ae06-9091be5196fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" Jul 22 11:54:47.471822 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:47.471788 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:47.742162 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:47.742060 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" Jul 22 11:54:47.948077 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:47.948042 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-29bk6"] Jul 22 11:54:47.951846 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:54:47.951799 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6bec41_3b3b_4cff_ae06_9091be5196fc.slice/crio-a41a859ad1e41add8b799eb05109de371cc630d20cef4c4fd9fb0def39a1fd96 WatchSource:0}: Error finding container a41a859ad1e41add8b799eb05109de371cc630d20cef4c4fd9fb0def39a1fd96: Status 404 returned error can't find the container with id a41a859ad1e41add8b799eb05109de371cc630d20cef4c4fd9fb0def39a1fd96 Jul 22 11:54:47.953575 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:47.953550 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 11:54:48.130927 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:48.130886 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" event={"ID":"db6bec41-3b3b-4cff-ae06-9091be5196fc","Type":"ContainerStarted","Data":"a41a859ad1e41add8b799eb05109de371cc630d20cef4c4fd9fb0def39a1fd96"} Jul 22 11:54:48.331271 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:48.331235 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-brfbg"] Jul 22 11:54:48.333773 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:48.333756 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" Jul 22 11:54:48.338805 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:54:48.338757 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"model-serving-api-dockercfg-kd9kj\" is forbidden: User \"system:node:ip-10-0-130-190.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-kd9kj\"" type="*v1.Secret" Jul 22 11:54:48.338805 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:54:48.338760 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"model-serving-api-54d89bd5fc-brfbg\" is forbidden: User \"system:node:ip-10-0-130-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.ec2.internal' and this object" podUID="1d911d66-0273-4aaa-b08b-264829357d6f" pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" Jul 22 11:54:48.338979 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:54:48.338876 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"model-serving-api-tls\" is forbidden: User \"system:node:ip-10-0-130-190.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" type="*v1.Secret" Jul 22 11:54:48.367647 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:48.367605 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-brfbg"] Jul 22 11:54:48.368891 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:48.368865 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d911d66-0273-4aaa-b08b-264829357d6f-tls-certs\") pod \"model-serving-api-54d89bd5fc-brfbg\" (UID: \"1d911d66-0273-4aaa-b08b-264829357d6f\") " pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" Jul 22 11:54:48.369053 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:48.368913 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stbz8\" (UniqueName: \"kubernetes.io/projected/1d911d66-0273-4aaa-b08b-264829357d6f-kube-api-access-stbz8\") pod \"model-serving-api-54d89bd5fc-brfbg\" (UID: \"1d911d66-0273-4aaa-b08b-264829357d6f\") " pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" Jul 22 11:54:48.377704 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:48.377666 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:48.470535 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:48.470429 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d911d66-0273-4aaa-b08b-264829357d6f-tls-certs\") pod \"model-serving-api-54d89bd5fc-brfbg\" (UID: \"1d911d66-0273-4aaa-b08b-264829357d6f\") " pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" Jul 22 11:54:48.470535 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:48.470484 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stbz8\" (UniqueName: \"kubernetes.io/projected/1d911d66-0273-4aaa-b08b-264829357d6f-kube-api-access-stbz8\") pod \"model-serving-api-54d89bd5fc-brfbg\" (UID: \"1d911d66-0273-4aaa-b08b-264829357d6f\") " pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" Jul 22 11:54:48.485016 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:48.484980 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stbz8\" (UniqueName: \"kubernetes.io/projected/1d911d66-0273-4aaa-b08b-264829357d6f-kube-api-access-stbz8\") pod \"model-serving-api-54d89bd5fc-brfbg\" (UID: \"1d911d66-0273-4aaa-b08b-264829357d6f\") " pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" Jul 22 11:54:49.301510 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:49.301469 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:49.471358 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:54:49.471286 2555 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: failed to sync secret cache: timed out waiting for the condition Jul 22 11:54:49.471820 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:54:49.471402 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d911d66-0273-4aaa-b08b-264829357d6f-tls-certs podName:1d911d66-0273-4aaa-b08b-264829357d6f nodeName:}" failed. No retries permitted until 2026-07-22 11:54:49.971378313 +0000 UTC m=+312.720589896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1d911d66-0273-4aaa-b08b-264829357d6f-tls-certs") pod "model-serving-api-54d89bd5fc-brfbg" (UID: "1d911d66-0273-4aaa-b08b-264829357d6f") : failed to sync secret cache: timed out waiting for the condition Jul 22 11:54:49.486184 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:49.486157 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-kd9kj\"" Jul 22 11:54:49.672638 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:49.672419 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 22 11:54:49.982364 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:49.982256 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d911d66-0273-4aaa-b08b-264829357d6f-tls-certs\") pod \"model-serving-api-54d89bd5fc-brfbg\" (UID: \"1d911d66-0273-4aaa-b08b-264829357d6f\") " pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" Jul 22 11:54:49.985246 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:49.985214 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d911d66-0273-4aaa-b08b-264829357d6f-tls-certs\") pod \"model-serving-api-54d89bd5fc-brfbg\" (UID: \"1d911d66-0273-4aaa-b08b-264829357d6f\") " pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" Jul 22 11:54:50.147462 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:50.147426 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" Jul 22 11:54:50.971171 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:50.971148 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-brfbg"] Jul 22 11:54:50.973464 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:54:50.973431 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d911d66_0273_4aaa_b08b_264829357d6f.slice/crio-1ed64b532e6ff5ea35c0b02f0a46b5052341b6f294d80c87cf9a5f1a02fc1125 WatchSource:0}: Error finding container 1ed64b532e6ff5ea35c0b02f0a46b5052341b6f294d80c87cf9a5f1a02fc1125: Status 404 returned error can't find the container with id 1ed64b532e6ff5ea35c0b02f0a46b5052341b6f294d80c87cf9a5f1a02fc1125 Jul 22 11:54:51.143216 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:51.143092 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" event={"ID":"db6bec41-3b3b-4cff-ae06-9091be5196fc","Type":"ContainerStarted","Data":"a0ed6e5835d156ce9d54dda2a540d29ceef06afb4a05e908951d8a4587667b35"} Jul 22 11:54:51.143371 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:51.143261 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" Jul 22 11:54:51.144262 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:51.144243 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" event={"ID":"1d911d66-0273-4aaa-b08b-264829357d6f","Type":"ContainerStarted","Data":"1ed64b532e6ff5ea35c0b02f0a46b5052341b6f294d80c87cf9a5f1a02fc1125"} Jul 22 11:54:51.182663 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:51.182603 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" podStartSLOduration=2.24458115 podStartE2EDuration="5.182580279s" totalImagesPullingTime="2.937999129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:54:51.180651423 +0000 UTC m=+313.929863026" watchObservedRunningTime="2026-07-22 11:54:51.182580279 +0000 UTC m=+313.931791881" Jul 22 11:54:53.152979 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:53.152947 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" event={"ID":"1d911d66-0273-4aaa-b08b-264829357d6f","Type":"ContainerStarted","Data":"44b94725381db6398349ab56673f5973466cfd6c7063ef929867c92e8338f316"} Jul 22 11:54:53.153343 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:53.153012 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" Jul 22 11:54:56.477876 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:56.477842 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:56.684756 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:56.684716 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:57.887103 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:57.887066 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:57.928269 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:57.928220 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" podStartSLOduration=7.942993559 podStartE2EDuration="9.928199912s" totalImagesPullingTime="1.985206353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:54:53.188379817 +0000 UTC m=+315.937591418" watchObservedRunningTime="2026-07-22 11:54:57.928199912 +0000 UTC m=+320.677411514" Jul 22 11:54:57.928602 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:57.928588 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57559476c9-vrvbz"] Jul 22 11:54:57.931419 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:57.931403 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" Jul 22 11:54:57.935727 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:57.935698 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 11:54:57.936558 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:57.936539 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 11:54:57.945429 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:57.945406 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-lhjjn\"" Jul 22 11:54:57.951218 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:57.951196 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 11:54:57.969073 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:57.969041 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 11:54:57.978624 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:57.978592 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57559476c9-vrvbz"] Jul 22 11:54:58.053397 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:58.053361 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/546779a6-4029-41ff-bedf-94bd239f0cbc-tmp\") pod \"kube-auth-proxy-57559476c9-vrvbz\" (UID: \"546779a6-4029-41ff-bedf-94bd239f0cbc\") " pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" Jul 22 11:54:58.053592 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:58.053424 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2cj\" (UniqueName: \"kubernetes.io/projected/546779a6-4029-41ff-bedf-94bd239f0cbc-kube-api-access-kp2cj\") pod \"kube-auth-proxy-57559476c9-vrvbz\" (UID: \"546779a6-4029-41ff-bedf-94bd239f0cbc\") " pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" Jul 22 11:54:58.053592 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:58.053476 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/546779a6-4029-41ff-bedf-94bd239f0cbc-tls-certs\") pod \"kube-auth-proxy-57559476c9-vrvbz\" (UID: \"546779a6-4029-41ff-bedf-94bd239f0cbc\") " pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" Jul 22 11:54:58.118231 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:58.118197 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:58.154680 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:58.154582 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/546779a6-4029-41ff-bedf-94bd239f0cbc-tmp\") pod \"kube-auth-proxy-57559476c9-vrvbz\" (UID: \"546779a6-4029-41ff-bedf-94bd239f0cbc\") " pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" Jul 22 11:54:58.154680 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:58.154628 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2cj\" (UniqueName: \"kubernetes.io/projected/546779a6-4029-41ff-bedf-94bd239f0cbc-kube-api-access-kp2cj\") pod \"kube-auth-proxy-57559476c9-vrvbz\" (UID: \"546779a6-4029-41ff-bedf-94bd239f0cbc\") " pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" Jul 22 11:54:58.154913 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:58.154757 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/546779a6-4029-41ff-bedf-94bd239f0cbc-tls-certs\") pod \"kube-auth-proxy-57559476c9-vrvbz\" (UID: \"546779a6-4029-41ff-bedf-94bd239f0cbc\") " pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" Jul 22 11:54:58.157018 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:58.156995 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/546779a6-4029-41ff-bedf-94bd239f0cbc-tmp\") pod \"kube-auth-proxy-57559476c9-vrvbz\" (UID: \"546779a6-4029-41ff-bedf-94bd239f0cbc\") " pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" Jul 22 11:54:58.157163 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:58.157148 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/546779a6-4029-41ff-bedf-94bd239f0cbc-tls-certs\") pod \"kube-auth-proxy-57559476c9-vrvbz\" (UID: \"546779a6-4029-41ff-bedf-94bd239f0cbc\") " pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" Jul 22 11:54:58.166575 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:58.166540 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2cj\" (UniqueName: \"kubernetes.io/projected/546779a6-4029-41ff-bedf-94bd239f0cbc-kube-api-access-kp2cj\") pod \"kube-auth-proxy-57559476c9-vrvbz\" (UID: \"546779a6-4029-41ff-bedf-94bd239f0cbc\") " pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" Jul 22 11:54:58.241351 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:58.241317 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" Jul 22 11:54:58.384321 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:58.384286 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57559476c9-vrvbz"] Jul 22 11:54:58.389001 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:54:58.388976 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546779a6_4029_41ff_bedf_94bd239f0cbc.slice/crio-343c85d794560fd00cad9998ba980d54b877cc72b8c8e68f486590e60c1b8126 WatchSource:0}: Error finding container 343c85d794560fd00cad9998ba980d54b877cc72b8c8e68f486590e60c1b8126: Status 404 returned error can't find the container with id 343c85d794560fd00cad9998ba980d54b877cc72b8c8e68f486590e60c1b8126 Jul 22 11:54:59.175486 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:54:59.175446 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" event={"ID":"546779a6-4029-41ff-bedf-94bd239f0cbc","Type":"ContainerStarted","Data":"343c85d794560fd00cad9998ba980d54b877cc72b8c8e68f486590e60c1b8126"} Jul 22 11:55:00.447635 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.447465 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:00.787250 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.787152 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:00.807364 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.807168 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-756sh"] Jul 22 11:55:00.811175 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.811147 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8"] Jul 22 11:55:00.811336 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.811212 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-756sh" Jul 22 11:55:00.814526 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.814499 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" Jul 22 11:55:00.821811 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.821789 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 22 11:55:00.823151 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.823041 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-wqx5c\"" Jul 22 11:55:00.823390 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.823323 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 22 11:55:00.823487 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.823472 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-56nzp\"" Jul 22 11:55:00.828397 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.828265 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8"] Jul 22 11:55:00.830612 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.830588 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-756sh"] Jul 22 11:55:00.877958 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.877919 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c291c1d6-703b-41c6-840d-cad9a4316686-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b9dk8\" (UID: \"c291c1d6-703b-41c6-840d-cad9a4316686\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" Jul 22 11:55:00.878169 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.877971 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bptfn\" (UniqueName: \"kubernetes.io/projected/5a4ed0bd-8ef3-4b59-8219-b82964e8fa13-kube-api-access-bptfn\") pod \"kserve-controller-manager-d6f4b65b8-756sh\" (UID: \"5a4ed0bd-8ef3-4b59-8219-b82964e8fa13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-756sh" Jul 22 11:55:00.878169 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.878010 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a4ed0bd-8ef3-4b59-8219-b82964e8fa13-cert\") pod \"kserve-controller-manager-d6f4b65b8-756sh\" (UID: \"5a4ed0bd-8ef3-4b59-8219-b82964e8fa13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-756sh" Jul 22 11:55:00.878169 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.878080 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bxsx\" (UniqueName: \"kubernetes.io/projected/c291c1d6-703b-41c6-840d-cad9a4316686-kube-api-access-8bxsx\") pod \"llmisvc-controller-manager-777c6bd7d5-b9dk8\" (UID: \"c291c1d6-703b-41c6-840d-cad9a4316686\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" Jul 22 11:55:00.884913 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.884892 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx"] Jul 22 11:55:00.888230 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.888211 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:00.895298 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.894863 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 11:55:00.895298 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.895178 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 11:55:00.896258 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.896134 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 11:55:00.896258 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.896157 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-x9zv5\"" Jul 22 11:55:00.896436 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.896363 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 11:55:00.910187 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.910156 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx"] Jul 22 11:55:00.963517 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.963396 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:00.979250 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.979214 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989d2dd9-95f1-4166-bd35-21c6214d665e-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:00.979431 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.979261 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bptfn\" (UniqueName: \"kubernetes.io/projected/5a4ed0bd-8ef3-4b59-8219-b82964e8fa13-kube-api-access-bptfn\") pod \"kserve-controller-manager-d6f4b65b8-756sh\" (UID: \"5a4ed0bd-8ef3-4b59-8219-b82964e8fa13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-756sh" Jul 22 11:55:00.979431 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.979280 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffcx\" (UniqueName: \"kubernetes.io/projected/989d2dd9-95f1-4166-bd35-21c6214d665e-kube-api-access-6ffcx\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:00.979431 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.979303 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/989d2dd9-95f1-4166-bd35-21c6214d665e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:00.979431 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.979326 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a4ed0bd-8ef3-4b59-8219-b82964e8fa13-cert\") pod \"kserve-controller-manager-d6f4b65b8-756sh\" (UID: \"5a4ed0bd-8ef3-4b59-8219-b82964e8fa13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-756sh" Jul 22 11:55:00.979431 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.979412 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bxsx\" (UniqueName: \"kubernetes.io/projected/c291c1d6-703b-41c6-840d-cad9a4316686-kube-api-access-8bxsx\") pod \"llmisvc-controller-manager-777c6bd7d5-b9dk8\" (UID: \"c291c1d6-703b-41c6-840d-cad9a4316686\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" Jul 22 11:55:00.979701 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.979444 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/989d2dd9-95f1-4166-bd35-21c6214d665e-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:00.979701 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.979466 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/989d2dd9-95f1-4166-bd35-21c6214d665e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:00.979701 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.979500 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c291c1d6-703b-41c6-840d-cad9a4316686-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b9dk8\" (UID: \"c291c1d6-703b-41c6-840d-cad9a4316686\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" Jul 22 11:55:00.979701 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.979520 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/989d2dd9-95f1-4166-bd35-21c6214d665e-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:00.979701 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.979543 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/989d2dd9-95f1-4166-bd35-21c6214d665e-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:00.979701 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:55:00.979687 2555 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 22 11:55:00.979957 ip-10-0-130-190 kubenswrapper[2555]: E0722 11:55:00.979818 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c291c1d6-703b-41c6-840d-cad9a4316686-cert podName:c291c1d6-703b-41c6-840d-cad9a4316686 nodeName:}" failed. No retries permitted until 2026-07-22 11:55:01.479797284 +0000 UTC m=+324.229008871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c291c1d6-703b-41c6-840d-cad9a4316686-cert") pod "llmisvc-controller-manager-777c6bd7d5-b9dk8" (UID: "c291c1d6-703b-41c6-840d-cad9a4316686") : secret "llmisvc-webhook-server-cert" not found Jul 22 11:55:00.981670 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.981650 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a4ed0bd-8ef3-4b59-8219-b82964e8fa13-cert\") pod \"kserve-controller-manager-d6f4b65b8-756sh\" (UID: \"5a4ed0bd-8ef3-4b59-8219-b82964e8fa13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-756sh" Jul 22 11:55:00.992339 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.992314 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bptfn\" (UniqueName: \"kubernetes.io/projected/5a4ed0bd-8ef3-4b59-8219-b82964e8fa13-kube-api-access-bptfn\") pod \"kserve-controller-manager-d6f4b65b8-756sh\" (UID: \"5a4ed0bd-8ef3-4b59-8219-b82964e8fa13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-756sh" Jul 22 11:55:00.995202 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:00.995180 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bxsx\" (UniqueName: \"kubernetes.io/projected/c291c1d6-703b-41c6-840d-cad9a4316686-kube-api-access-8bxsx\") pod \"llmisvc-controller-manager-777c6bd7d5-b9dk8\" (UID: \"c291c1d6-703b-41c6-840d-cad9a4316686\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" Jul 22 11:55:01.080313 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.080223 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989d2dd9-95f1-4166-bd35-21c6214d665e-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.080313 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.080262 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffcx\" (UniqueName: \"kubernetes.io/projected/989d2dd9-95f1-4166-bd35-21c6214d665e-kube-api-access-6ffcx\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.080313 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.080286 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/989d2dd9-95f1-4166-bd35-21c6214d665e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.080593 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.080346 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/989d2dd9-95f1-4166-bd35-21c6214d665e-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.080593 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.080373 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/989d2dd9-95f1-4166-bd35-21c6214d665e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.080593 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.080511 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/989d2dd9-95f1-4166-bd35-21c6214d665e-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.080593 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.080551 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/989d2dd9-95f1-4166-bd35-21c6214d665e-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.081014 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.080996 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/989d2dd9-95f1-4166-bd35-21c6214d665e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.082765 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.082722 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/989d2dd9-95f1-4166-bd35-21c6214d665e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.082882 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.082787 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/989d2dd9-95f1-4166-bd35-21c6214d665e-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.083002 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.082982 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/989d2dd9-95f1-4166-bd35-21c6214d665e-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.083157 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.083140 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/989d2dd9-95f1-4166-bd35-21c6214d665e-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.096756 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.096728 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989d2dd9-95f1-4166-bd35-21c6214d665e-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.096934 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.096915 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffcx\" (UniqueName: \"kubernetes.io/projected/989d2dd9-95f1-4166-bd35-21c6214d665e-kube-api-access-6ffcx\") pod \"istiod-openshift-gateway-94bb8fbfd-2rjcx\" (UID: \"989d2dd9-95f1-4166-bd35-21c6214d665e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.114931 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.114900 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:01.126837 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.126817 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-756sh" Jul 22 11:55:01.186844 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.186805 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" event={"ID":"546779a6-4029-41ff-bedf-94bd239f0cbc","Type":"ContainerStarted","Data":"65d9abb27d5d9c5c0c7efccdd18ccb2f1a0248aa95773bbfe63bbee2139b6d42"} Jul 22 11:55:01.198876 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.198845 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:01.221463 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.221399 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57559476c9-vrvbz" podStartSLOduration=1.726637599 podStartE2EDuration="4.221375365s" totalImagesPullingTime="2.494737766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:55:01.220685355 +0000 UTC m=+323.969896957" watchObservedRunningTime="2026-07-22 11:55:01.221375365 +0000 UTC m=+323.970586968" Jul 22 11:55:01.278478 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.278424 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:01.366175 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.366137 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-756sh"] Jul 22 11:55:01.370590 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:55:01.370563 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4ed0bd_8ef3_4b59_8219_b82964e8fa13.slice/crio-971a68d9f46543be498cbeb352d93139febe2a38bd4b9ac2c9f2fe0cde86d29f WatchSource:0}: Error finding container 971a68d9f46543be498cbeb352d93139febe2a38bd4b9ac2c9f2fe0cde86d29f: Status 404 returned error can't find the container with id 971a68d9f46543be498cbeb352d93139febe2a38bd4b9ac2c9f2fe0cde86d29f Jul 22 11:55:01.484752 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.484715 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c291c1d6-703b-41c6-840d-cad9a4316686-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b9dk8\" (UID: \"c291c1d6-703b-41c6-840d-cad9a4316686\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" Jul 22 11:55:01.487033 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.487015 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c291c1d6-703b-41c6-840d-cad9a4316686-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b9dk8\" (UID: \"c291c1d6-703b-41c6-840d-cad9a4316686\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" Jul 22 11:55:01.491179 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.491157 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx"] Jul 22 11:55:01.502262 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:55:01.502230 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989d2dd9_95f1_4166_bd35_21c6214d665e.slice/crio-1bf95fb1ce748016ba4ea0a2046ed0ccf19b91929f55a153fb87129fa9ab7cb8 WatchSource:0}: Error finding container 1bf95fb1ce748016ba4ea0a2046ed0ccf19b91929f55a153fb87129fa9ab7cb8: Status 404 returned error can't find the container with id 1bf95fb1ce748016ba4ea0a2046ed0ccf19b91929f55a153fb87129fa9ab7cb8 Jul 22 11:55:01.735089 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.734997 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" Jul 22 11:55:01.944331 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:01.944251 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8"] Jul 22 11:55:01.947872 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:55:01.947831 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc291c1d6_703b_41c6_840d_cad9a4316686.slice/crio-1f4b526b2ef5a6478c6d3bec7c77eb9bf42dd8d92be3f69956b3360ca1cd90a6 WatchSource:0}: Error finding container 1f4b526b2ef5a6478c6d3bec7c77eb9bf42dd8d92be3f69956b3360ca1cd90a6: Status 404 returned error can't find the container with id 1f4b526b2ef5a6478c6d3bec7c77eb9bf42dd8d92be3f69956b3360ca1cd90a6 Jul 22 11:55:02.138828 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:02.138798 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:02.150476 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:02.150450 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-29bk6" Jul 22 11:55:02.199504 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:02.199464 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-756sh" event={"ID":"5a4ed0bd-8ef3-4b59-8219-b82964e8fa13","Type":"ContainerStarted","Data":"971a68d9f46543be498cbeb352d93139febe2a38bd4b9ac2c9f2fe0cde86d29f"} Jul 22 11:55:02.200716 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:02.200684 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" event={"ID":"989d2dd9-95f1-4166-bd35-21c6214d665e","Type":"ContainerStarted","Data":"1bf95fb1ce748016ba4ea0a2046ed0ccf19b91929f55a153fb87129fa9ab7cb8"} Jul 22 11:55:02.201817 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:02.201797 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" event={"ID":"c291c1d6-703b-41c6-840d-cad9a4316686","Type":"ContainerStarted","Data":"1f4b526b2ef5a6478c6d3bec7c77eb9bf42dd8d92be3f69956b3360ca1cd90a6"} Jul 22 11:55:02.295166 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:02.295131 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:02.813141 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:02.813092 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:04.161905 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:04.161881 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-brfbg" Jul 22 11:55:06.399407 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:06.399368 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 11:55:06.399803 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:06.399433 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 11:55:07.229770 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:07.229735 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-756sh" event={"ID":"5a4ed0bd-8ef3-4b59-8219-b82964e8fa13","Type":"ContainerStarted","Data":"dcf60e91d53f14396eb97ce478997524fed093c4402c8a87ddc3c34188aba1b8"} Jul 22 11:55:07.229966 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:07.229874 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-756sh" Jul 22 11:55:07.231173 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:07.231147 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" event={"ID":"989d2dd9-95f1-4166-bd35-21c6214d665e","Type":"ContainerStarted","Data":"d859782ca509453a44495c524e0673ac23f8add022d9267b4af2f7501fbeb417"} Jul 22 11:55:07.231287 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:07.231193 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:07.232714 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:07.232682 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" event={"ID":"c291c1d6-703b-41c6-840d-cad9a4316686","Type":"ContainerStarted","Data":"67b3d8ad1f518c00c23b19281dc56d9243a0c8c8d6f391969ce63a6d4a642f13"} Jul 22 11:55:07.232886 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:07.232873 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" Jul 22 11:55:07.233157 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:07.233135 2555 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-2rjcx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 11:55:07.233217 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:07.233185 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" podUID="989d2dd9-95f1-4166-bd35-21c6214d665e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 11:55:07.252796 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:07.252739 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-756sh" podStartSLOduration=2.23779774 podStartE2EDuration="7.252721881s" totalImagesPullingTime="5.014924141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:55:07.251448201 +0000 UTC m=+330.000659807" watchObservedRunningTime="2026-07-22 11:55:07.252721881 +0000 UTC m=+330.001933482" Jul 22 11:55:07.275540 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:07.275473 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" podStartSLOduration=2.83160897 podStartE2EDuration="7.27545063s" totalImagesPullingTime="4.44384166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:55:07.273129424 +0000 UTC m=+330.022341024" watchObservedRunningTime="2026-07-22 11:55:07.27545063 +0000 UTC m=+330.024662233" Jul 22 11:55:07.310900 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:07.310825 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" podStartSLOduration=2.415799276 podStartE2EDuration="7.310791624s" totalImagesPullingTime="4.894992348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:55:07.308893362 +0000 UTC m=+330.058104965" watchObservedRunningTime="2026-07-22 11:55:07.310791624 +0000 UTC m=+330.060003226" Jul 22 11:55:07.367570 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:07.367538 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:07.555798 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:07.555709 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:08.239257 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:08.239232 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2rjcx" Jul 22 11:55:08.661867 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:08.661830 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:09.243126 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:09.243077 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:38.240773 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:38.240684 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9dk8" Jul 22 11:55:38.243716 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:38.243698 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-756sh" Jul 22 11:55:38.845231 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:38.845188 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:38.946001 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:38.945961 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:47.467200 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:47.467157 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:47.562791 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:47.562757 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:47.675968 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:47.675931 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:49.087290 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:55:49.087252 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:08.273163 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:08.273100 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:10.658693 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:10.658658 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:14.102313 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:14.102267 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:19.767809 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:19.767772 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hr6xw"] Jul 22 11:56:19.769841 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:19.769824 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hr6xw" Jul 22 11:56:19.779449 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:19.779428 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 11:56:19.779564 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:19.779484 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 11:56:19.782200 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:19.782185 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-52grc\"" Jul 22 11:56:19.795779 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:19.795751 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hr6xw"] Jul 22 11:56:19.940550 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:19.940508 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kszp5\" (UniqueName: \"kubernetes.io/projected/c870d06d-a4b3-452b-9de1-f3bd939abe36-kube-api-access-kszp5\") pod \"authorino-operator-75cc8d5cb7-hr6xw\" (UID: \"c870d06d-a4b3-452b-9de1-f3bd939abe36\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hr6xw" Jul 22 11:56:20.041559 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:20.041476 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kszp5\" (UniqueName: \"kubernetes.io/projected/c870d06d-a4b3-452b-9de1-f3bd939abe36-kube-api-access-kszp5\") pod \"authorino-operator-75cc8d5cb7-hr6xw\" (UID: \"c870d06d-a4b3-452b-9de1-f3bd939abe36\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hr6xw" Jul 22 11:56:20.051898 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:20.051863 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kszp5\" (UniqueName: \"kubernetes.io/projected/c870d06d-a4b3-452b-9de1-f3bd939abe36-kube-api-access-kszp5\") pod \"authorino-operator-75cc8d5cb7-hr6xw\" (UID: \"c870d06d-a4b3-452b-9de1-f3bd939abe36\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hr6xw" Jul 22 11:56:20.080003 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:20.079951 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hr6xw" Jul 22 11:56:20.217448 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:20.217383 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hr6xw"] Jul 22 11:56:20.220142 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:56:20.220099 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc870d06d_a4b3_452b_9de1_f3bd939abe36.slice/crio-5a45ebc35690d3f44bbc5569ccda53eb7880e2970cbeed4986e113de8351cff7 WatchSource:0}: Error finding container 5a45ebc35690d3f44bbc5569ccda53eb7880e2970cbeed4986e113de8351cff7: Status 404 returned error can't find the container with id 5a45ebc35690d3f44bbc5569ccda53eb7880e2970cbeed4986e113de8351cff7 Jul 22 11:56:20.523673 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:20.523639 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hr6xw" event={"ID":"c870d06d-a4b3-452b-9de1-f3bd939abe36","Type":"ContainerStarted","Data":"5a45ebc35690d3f44bbc5569ccda53eb7880e2970cbeed4986e113de8351cff7"} Jul 22 11:56:28.558213 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:28.558174 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hr6xw" event={"ID":"c870d06d-a4b3-452b-9de1-f3bd939abe36","Type":"ContainerStarted","Data":"0d3c939063de604602852e0fc16f751cf632752153968283343c67ec4b149fec"} Jul 22 11:56:28.558664 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:28.558276 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hr6xw" Jul 22 11:56:28.579825 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:28.579775 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hr6xw" podStartSLOduration=2.116906854 podStartE2EDuration="9.579761017s" totalImagesPullingTime="7.462854163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:56:28.579013921 +0000 UTC m=+411.328225524" watchObservedRunningTime="2026-07-22 11:56:28.579761017 +0000 UTC m=+411.328972618" Jul 22 11:56:31.152860 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:31.152828 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:31.234787 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:31.234753 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:33.246533 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:33.246500 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:33.346851 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:33.346814 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:38.120563 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:38.120528 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:38.216499 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:38.216463 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:39.564998 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:39.564970 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hr6xw" Jul 22 11:56:40.336785 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:40.336742 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:40.555222 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:40.555188 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:40.752243 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:40.752156 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:46.050941 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:46.050907 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:46.152953 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:46.152920 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:47.620461 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:47.620427 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:47.714662 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:47.714624 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:49.706933 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:49.706899 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:52.956318 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:52.956283 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:54.886814 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:54.886777 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:57.755249 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:57.755213 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:57.858698 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:56:57.858661 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:00.305423 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:00.305342 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:00.466609 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:00.466569 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:18.257296 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:18.257254 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:29.965604 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:29.965565 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:30.086360 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:30.086328 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:31.928548 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:31.928517 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:32.237222 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:32.237133 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:44.324430 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.324394 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:44.423986 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.423953 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:44.531212 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.531182 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:44.770352 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.770314 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 11:57:44.773574 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.773556 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-mr49t" Jul 22 11:57:44.776589 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.776568 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 11:57:44.776723 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.776636 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z7z59\"" Jul 22 11:57:44.785078 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.785055 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 11:57:44.814242 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.814213 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 11:57:44.825046 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.825021 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ef3f9e8-b94c-476f-a86a-12c28c9e7741-config-file\") pod \"limitador-limitador-54d6d988d5-mr49t\" (UID: \"3ef3f9e8-b94c-476f-a86a-12c28c9e7741\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mr49t" Jul 22 11:57:44.825194 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.825055 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98ln9\" (UniqueName: \"kubernetes.io/projected/3ef3f9e8-b94c-476f-a86a-12c28c9e7741-kube-api-access-98ln9\") pod \"limitador-limitador-54d6d988d5-mr49t\" (UID: \"3ef3f9e8-b94c-476f-a86a-12c28c9e7741\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mr49t" Jul 22 11:57:44.926185 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.926144 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ef3f9e8-b94c-476f-a86a-12c28c9e7741-config-file\") pod \"limitador-limitador-54d6d988d5-mr49t\" (UID: \"3ef3f9e8-b94c-476f-a86a-12c28c9e7741\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mr49t" Jul 22 11:57:44.926185 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.926183 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98ln9\" (UniqueName: \"kubernetes.io/projected/3ef3f9e8-b94c-476f-a86a-12c28c9e7741-kube-api-access-98ln9\") pod \"limitador-limitador-54d6d988d5-mr49t\" (UID: \"3ef3f9e8-b94c-476f-a86a-12c28c9e7741\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mr49t" Jul 22 11:57:44.926821 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.926801 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ef3f9e8-b94c-476f-a86a-12c28c9e7741-config-file\") pod \"limitador-limitador-54d6d988d5-mr49t\" (UID: \"3ef3f9e8-b94c-476f-a86a-12c28c9e7741\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mr49t" Jul 22 11:57:44.937456 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:44.937424 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98ln9\" (UniqueName: \"kubernetes.io/projected/3ef3f9e8-b94c-476f-a86a-12c28c9e7741-kube-api-access-98ln9\") pod \"limitador-limitador-54d6d988d5-mr49t\" (UID: \"3ef3f9e8-b94c-476f-a86a-12c28c9e7741\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mr49t" Jul 22 11:57:45.084199 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:45.084166 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-mr49t" Jul 22 11:57:45.223375 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:45.223352 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 11:57:45.225681 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:57:45.225647 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef3f9e8_b94c_476f_a86a_12c28c9e7741.slice/crio-c45e3a5291ed706cbcc190b6bbadfa9a471f880eff65262e59e838e4e0dd8384 WatchSource:0}: Error finding container c45e3a5291ed706cbcc190b6bbadfa9a471f880eff65262e59e838e4e0dd8384: Status 404 returned error can't find the container with id c45e3a5291ed706cbcc190b6bbadfa9a471f880eff65262e59e838e4e0dd8384 Jul 22 11:57:45.261790 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:45.261750 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:45.352419 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:45.352319 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:45.877635 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:45.877538 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-mr49t" event={"ID":"3ef3f9e8-b94c-476f-a86a-12c28c9e7741","Type":"ContainerStarted","Data":"c45e3a5291ed706cbcc190b6bbadfa9a471f880eff65262e59e838e4e0dd8384"} Jul 22 11:57:47.887179 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:47.887146 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-mr49t" event={"ID":"3ef3f9e8-b94c-476f-a86a-12c28c9e7741","Type":"ContainerStarted","Data":"32423dae7464b7dc4a17cd7364e3aa2ec57a67622eb39e995a3e3ed095174566"} Jul 22 11:57:47.887535 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:47.887213 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-mr49t" Jul 22 11:57:47.905792 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:47.905751 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-mr49t" podStartSLOduration=1.324271999 podStartE2EDuration="3.905735951s" totalImagesPullingTime="2.581463952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:57:47.905161854 +0000 UTC m=+490.654373456" watchObservedRunningTime="2026-07-22 11:57:47.905735951 +0000 UTC m=+490.654947552" Jul 22 11:57:50.971170 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:50.971137 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:51.055302 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:51.055266 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:58.893302 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:57:58.893271 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-mr49t" Jul 22 11:58:01.904621 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:58:01.904585 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:02.005575 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:58:02.005522 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:02.259262 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:58:02.259181 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:05.171009 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:58:05.170980 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:05.272184 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:58:05.272140 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:05.683364 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:58:05.683333 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:18.206591 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:58:18.206560 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:18.557043 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:58:18.556947 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:25.992539 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:58:25.992503 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:26.121497 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:58:26.121454 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:29.430915 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:58:29.430822 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:29.621406 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:58:29.621365 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:00.412300 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:00.412265 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:01.524013 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:01.523978 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:01.621045 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:01.621012 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:02.120452 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:02.120418 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:02.449532 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:02.449427 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:02.811754 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:02.811652 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:02.921170 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:02.921136 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:03.063710 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:03.063622 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 11:59:12.874652 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:12.874615 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:13.019441 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:13.019403 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:19.382469 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:19.382430 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:19.906242 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:19.906205 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 11:59:24.258228 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:24.258187 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:25.092649 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:25.092610 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 11:59:28.856617 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:28.856578 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:30.124189 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:30.124154 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 11:59:30.445289 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:30.445199 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:31.409681 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:31.409643 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:34.011268 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:34.011229 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:34.249001 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:34.248968 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:42.148799 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:42.148755 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:43.777153 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:43.777103 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 11:59:50.724151 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.724101 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6"] Jul 22 11:59:50.727349 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.727332 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.730321 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.730301 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 11:59:50.730853 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.730839 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 22 11:59:50.731325 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.731305 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 11:59:50.731747 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.731734 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rx8m7\"" Jul 22 11:59:50.741361 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.741336 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6"] Jul 22 11:59:50.747691 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.747667 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5177e6a9-f007-499f-8d67-aefed6851e13-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.747792 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.747729 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5177e6a9-f007-499f-8d67-aefed6851e13-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.747792 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.747763 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5177e6a9-f007-499f-8d67-aefed6851e13-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.747883 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.747804 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6pnk\" (UniqueName: \"kubernetes.io/projected/5177e6a9-f007-499f-8d67-aefed6851e13-kube-api-access-c6pnk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.747883 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.747847 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5177e6a9-f007-499f-8d67-aefed6851e13-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.747956 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.747919 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5177e6a9-f007-499f-8d67-aefed6851e13-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.770428 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.770403 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:50.848588 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.848555 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5177e6a9-f007-499f-8d67-aefed6851e13-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.848770 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.848594 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5177e6a9-f007-499f-8d67-aefed6851e13-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.848770 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.848636 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6pnk\" (UniqueName: \"kubernetes.io/projected/5177e6a9-f007-499f-8d67-aefed6851e13-kube-api-access-c6pnk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.848770 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.848673 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5177e6a9-f007-499f-8d67-aefed6851e13-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.848770 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.848706 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5177e6a9-f007-499f-8d67-aefed6851e13-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.848770 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.848756 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5177e6a9-f007-499f-8d67-aefed6851e13-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.848995 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.848967 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5177e6a9-f007-499f-8d67-aefed6851e13-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.849089 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.849068 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5177e6a9-f007-499f-8d67-aefed6851e13-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.849208 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.849190 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5177e6a9-f007-499f-8d67-aefed6851e13-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.851024 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.850997 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5177e6a9-f007-499f-8d67-aefed6851e13-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.851190 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.851171 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5177e6a9-f007-499f-8d67-aefed6851e13-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:50.865656 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:50.865628 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6pnk\" (UniqueName: \"kubernetes.io/projected/5177e6a9-f007-499f-8d67-aefed6851e13-kube-api-access-c6pnk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdvr6\" (UID: \"5177e6a9-f007-499f-8d67-aefed6851e13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:51.037410 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:51.037309 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 11:59:51.179748 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:51.179646 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6"] Jul 22 11:59:51.182454 ip-10-0-130-190 kubenswrapper[2555]: W0722 11:59:51.182424 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5177e6a9_f007_499f_8d67_aefed6851e13.slice/crio-147a023269359022d808beb20997fa7fa8e58a8be92e161f0fa680363d56e0a5 WatchSource:0}: Error finding container 147a023269359022d808beb20997fa7fa8e58a8be92e161f0fa680363d56e0a5: Status 404 returned error can't find the container with id 147a023269359022d808beb20997fa7fa8e58a8be92e161f0fa680363d56e0a5 Jul 22 11:59:51.183789 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:51.183771 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 11:59:51.395508 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:51.395475 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" event={"ID":"5177e6a9-f007-499f-8d67-aefed6851e13","Type":"ContainerStarted","Data":"147a023269359022d808beb20997fa7fa8e58a8be92e161f0fa680363d56e0a5"} Jul 22 11:59:52.401311 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:52.401280 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" event={"ID":"5177e6a9-f007-499f-8d67-aefed6851e13","Type":"ContainerStarted","Data":"fd43febfd2a857a0b02b83c93cf29bbd5e3ec91541f2af8532658afba50486a2"} Jul 22 11:59:52.423900 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:52.423846 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" podStartSLOduration=1.294594939 podStartE2EDuration="2.42382838s" totalImagesPullingTime="1.129233441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:59:52.423551163 +0000 UTC m=+615.172762765" watchObservedRunningTime="2026-07-22 11:59:52.42382838 +0000 UTC m=+615.173039973" Jul 22 11:59:53.386944 ip-10-0-130-190 kubenswrapper[2555]: I0722 11:59:53.386915 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:00:01.038247 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:00:01.038149 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 12:00:01.038247 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:00:01.038220 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 12:00:01.050723 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:00:01.050699 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 12:00:01.449424 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:00:01.449396 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdvr6" Jul 22 12:00:01.982961 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:00:01.982930 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:05.612380 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:00:05.608644 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:00:48.047609 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:00:48.047570 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:48.176918 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:00:48.176877 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:49.332950 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:00:49.332903 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:49.518255 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:00:49.518223 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:14.886429 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:01:14.886387 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:01:26.625688 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:01:26.625648 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:26.748512 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:01:26.748481 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:27.600563 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:01:27.600531 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:01:35.502573 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:01:35.502493 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:01:37.659486 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:01:37.659452 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:46.879086 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:01:46.879048 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:01:52.998636 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:01:52.998606 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:53.099053 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:01:53.098995 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:02:59.789347 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:02:59.789251 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:01.601024 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:03:01.600988 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:25.782030 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:03:25.781990 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:03:38.987625 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:03:38.987592 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:03:43.608900 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:03:43.608862 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:43.712424 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:03:43.712386 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:43.869645 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:03:43.869557 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:51.281810 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:03:51.281771 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:51.388744 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:03:51.388662 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:51.484308 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:03:51.484276 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:51.634021 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:03:51.633989 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:57.950275 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:03:57.950245 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:58.028086 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:03:58.028056 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:09.194761 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:04:09.194726 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:14.983842 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:04:14.983806 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:04:18.262884 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:04:18.262850 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:24.387816 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:04:24.387771 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:04:33.386262 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:04:33.386177 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:04:44.284832 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:04:44.284795 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:04:54.100050 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:04:54.100012 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:05:06.283339 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:05:06.283307 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:05:16.685034 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:05:16.684994 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:05:26.677618 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:05:26.677583 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:05:51.990066 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:05:51.990027 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:06:03.398567 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:06:03.398485 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:06:27.889385 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:06:27.889352 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:06:34.878773 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:06:34.878734 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:06:39.085124 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:06:39.085082 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:06:53.487274 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:06:53.487240 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:07:04.781752 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:07:04.781722 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:07:14.524620 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:07:14.524582 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:07:24.026005 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:07:24.025961 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:07:33.478083 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:07:33.477998 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:08:54.321230 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:08:54.321194 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:09:20.813054 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:09:20.812971 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:09:30.174184 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:09:30.174152 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:09:42.995319 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:09:42.995285 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:09:55.808768 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:09:55.808727 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:10:05.208356 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:10:05.208317 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:10:16.972730 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:10:16.972697 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:10:25.673318 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:10:25.673281 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:10:35.418249 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:10:35.418156 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:10:47.876704 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:10:47.876672 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:10:55.772082 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:10:55.772044 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:11:06.374136 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:11:06.374083 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:11:21.513437 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:11:21.513399 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:27.976355 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:11:27.976315 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:11:37.475407 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:11:37.475367 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:11:42.440005 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:11:42.439965 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:47.079996 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:11:47.079957 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:12:00.516319 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:12:00.516241 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:12:03.883228 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:12:03.883192 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:12:33.337435 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:12:33.337395 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:12:34.353333 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:12:34.353295 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:12:35.280265 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:12:35.280228 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:12:47.477403 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:12:47.477365 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:12:54.172204 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:12:54.172167 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:13:09.978487 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:13:09.978455 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:13:15.412079 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:13:15.412044 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:13:15.511952 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:13:15.511918 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:13:20.368480 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:13:20.368446 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:13:29.578190 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:13:29.578092 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:13:39.885163 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:13:39.885132 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:13:49.266334 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:13:49.266292 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:13:58.375830 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:13:58.375794 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:14:08.599945 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:14:08.599910 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:14:08.706551 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:14:08.706517 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:14:12.314801 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:14:12.314764 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:14:21.472141 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:14:21.472091 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:14:31.175290 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:14:31.175254 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:14:40.471861 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:14:40.471823 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:14:53.370453 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:14:53.370422 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:15:01.578097 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:15:01.578011 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:15:13.586753 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:15:13.586713 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:15:25.477662 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:15:25.477624 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:15:36.510546 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:15:36.510509 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:15:38.964408 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:15:38.964369 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:15:49.375880 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:15:49.375846 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:15:58.675604 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:15:58.675564 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:15:59.425774 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:15:59.425735 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:02.491614 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:02.491579 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:02.595727 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:02.595694 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:02.942077 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:02.942038 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:08.886834 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:08.886798 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:16:10.036096 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:10.036063 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:10.134768 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:10.134735 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:10.660390 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:10.660355 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:10.757919 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:10.757883 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:10.858002 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:10.857969 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:13.368341 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:13.368292 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:13.552346 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:13.552303 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:13.653309 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:13.653231 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:14.057771 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:14.057687 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:14.160486 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:14.160457 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:18.576444 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:18.576407 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:16:49.607937 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:49.607841 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:52.478267 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:52.478232 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:16:54.017654 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:54.017618 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:54.218701 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:54.218655 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:54.426864 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:54.426825 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:54.676129 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:54.676083 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:54.775678 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:54.775589 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:54.910687 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:54.910641 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:55.253180 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:55.253142 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:55.414486 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:55.414451 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:57.747213 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:57.747186 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:57.868760 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:57.868721 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:57.976382 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:57.976342 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:58.367724 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:58.367687 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:16:58.467691 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:16:58.467656 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:03.283899 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:03.283864 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:17:33.242039 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:33.241998 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:36.165169 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:36.165129 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:17:37.924698 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:37.924655 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:38.082920 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:38.082872 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:38.306841 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:38.306761 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:38.546453 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:38.546414 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:39.073903 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:39.073866 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:39.174878 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:39.174810 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:40.830907 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:40.830872 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:40.930654 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:40.930616 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:41.080799 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:41.080748 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:41.178981 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:41.178940 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:41.787992 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:41.787960 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:41.887446 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:41.887410 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:17:47.172753 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:17:47.172721 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:18:06.743675 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:06.743587 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69"] Jul 22 12:18:06.747012 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:06.746995 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69" Jul 22 12:18:06.749513 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:06.749489 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 12:18:06.749644 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:06.749494 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kmhv\"" Jul 22 12:18:06.754440 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:06.754054 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69"] Jul 22 12:18:06.870580 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:06.870542 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3835b854-e8e9-4e1d-82f9-8b26df3c011f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69\" (UID: \"3835b854-e8e9-4e1d-82f9-8b26df3c011f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69" Jul 22 12:18:06.971830 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:06.971801 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3835b854-e8e9-4e1d-82f9-8b26df3c011f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69\" (UID: \"3835b854-e8e9-4e1d-82f9-8b26df3c011f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69" Jul 22 12:18:06.972440 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:06.972421 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3835b854-e8e9-4e1d-82f9-8b26df3c011f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69\" (UID: \"3835b854-e8e9-4e1d-82f9-8b26df3c011f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69" Jul 22 12:18:07.059095 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:07.059014 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69" Jul 22 12:18:07.188158 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:07.188128 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69"] Jul 22 12:18:07.191142 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:18:07.191090 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3835b854_e8e9_4e1d_82f9_8b26df3c011f.slice/crio-a158b6f30b0a7dfd7dc0f6ce8939e3e4adca16ec1c6b2553409bd81f753dda4f WatchSource:0}: Error finding container a158b6f30b0a7dfd7dc0f6ce8939e3e4adca16ec1c6b2553409bd81f753dda4f: Status 404 returned error can't find the container with id a158b6f30b0a7dfd7dc0f6ce8939e3e4adca16ec1c6b2553409bd81f753dda4f Jul 22 12:18:07.193044 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:07.193025 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 12:18:07.914629 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:07.914595 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69" event={"ID":"3835b854-e8e9-4e1d-82f9-8b26df3c011f","Type":"ContainerStarted","Data":"a158b6f30b0a7dfd7dc0f6ce8939e3e4adca16ec1c6b2553409bd81f753dda4f"} Jul 22 12:18:09.924584 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:09.924552 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69" event={"ID":"3835b854-e8e9-4e1d-82f9-8b26df3c011f","Type":"ContainerStarted","Data":"a644b8becfe130b550e1aafb989f16a0287d825f92d50a4e96395ee2bf81eb2a"} Jul 22 12:18:09.940909 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:09.940858 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69" podStartSLOduration=1.706389325 podStartE2EDuration="3.940840951s" totalImagesPullingTime="2.234451626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:18:09.94008387 +0000 UTC m=+1712.689295473" watchObservedRunningTime="2026-07-22 12:18:09.940840951 +0000 UTC m=+1712.690052553" Jul 22 12:18:10.929658 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:10.929622 2555 generic.go:356] "Generic (PLEG): container finished" podID="3835b854-e8e9-4e1d-82f9-8b26df3c011f" containerID="a644b8becfe130b550e1aafb989f16a0287d825f92d50a4e96395ee2bf81eb2a" exitCode=0 Jul 22 12:18:10.930133 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:10.929694 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69" event={"ID":"3835b854-e8e9-4e1d-82f9-8b26df3c011f","Type":"ContainerDied","Data":"a644b8becfe130b550e1aafb989f16a0287d825f92d50a4e96395ee2bf81eb2a"} Jul 22 12:18:12.070415 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:12.070388 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69" Jul 22 12:18:12.216890 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:12.216802 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3835b854-e8e9-4e1d-82f9-8b26df3c011f-maas-api-service-ca\") pod \"3835b854-e8e9-4e1d-82f9-8b26df3c011f\" (UID: \"3835b854-e8e9-4e1d-82f9-8b26df3c011f\") " Jul 22 12:18:12.217237 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:12.217215 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3835b854-e8e9-4e1d-82f9-8b26df3c011f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3835b854-e8e9-4e1d-82f9-8b26df3c011f" (UID: "3835b854-e8e9-4e1d-82f9-8b26df3c011f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:18:12.317501 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:12.317468 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3835b854-e8e9-4e1d-82f9-8b26df3c011f-maas-api-service-ca\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:18:12.939716 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:12.939682 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69" event={"ID":"3835b854-e8e9-4e1d-82f9-8b26df3c011f","Type":"ContainerDied","Data":"a158b6f30b0a7dfd7dc0f6ce8939e3e4adca16ec1c6b2553409bd81f753dda4f"} Jul 22 12:18:12.939716 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:12.939716 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69" Jul 22 12:18:12.939921 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:12.939731 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a158b6f30b0a7dfd7dc0f6ce8939e3e4adca16ec1c6b2553409bd81f753dda4f" Jul 22 12:18:17.235453 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:17.235413 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:20.071812 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:20.071775 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:18:21.830780 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:21.830749 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69"] Jul 22 12:18:21.831181 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:21.830748 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:21.833166 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:21.833143 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-800b6cf7-jsp69"] Jul 22 12:18:21.958437 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:21.958395 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3835b854-e8e9-4e1d-82f9-8b26df3c011f" path="/var/lib/kubelet/pods/3835b854-e8e9-4e1d-82f9-8b26df3c011f/volumes" Jul 22 12:18:23.013850 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:23.013812 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:23.225423 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:23.225381 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:23.444866 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:23.444828 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:23.756478 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:23.756394 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:23.999439 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:23.999403 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:24.180903 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:24.180866 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:24.284474 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:24.284439 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:26.107617 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:26.107583 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:26.196022 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:26.195978 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:26.267068 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:26.267025 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:26.367488 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:26.367413 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:26.722016 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:26.721923 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:26.819346 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:26.819313 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:18:28.870950 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:28.870913 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:18:50.997051 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:50.997016 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc"] Jul 22 12:18:50.997581 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:50.997359 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3835b854-e8e9-4e1d-82f9-8b26df3c011f" containerName="revoke-keys" Jul 22 12:18:50.997581 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:50.997370 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3835b854-e8e9-4e1d-82f9-8b26df3c011f" containerName="revoke-keys" Jul 22 12:18:50.997581 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:50.997433 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3835b854-e8e9-4e1d-82f9-8b26df3c011f" containerName="revoke-keys" Jul 22 12:18:51.000331 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:51.000316 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc" Jul 22 12:18:51.003055 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:51.003033 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 12:18:51.003206 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:51.003057 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kmhv\"" Jul 22 12:18:51.008688 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:51.008647 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc"] Jul 22 12:18:51.034265 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:51.034229 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c909037b-917d-4188-aeb1-9a728654092f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-94d621ea-gshfc\" (UID: \"c909037b-917d-4188-aeb1-9a728654092f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc" Jul 22 12:18:51.135462 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:51.135429 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c909037b-917d-4188-aeb1-9a728654092f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-94d621ea-gshfc\" (UID: \"c909037b-917d-4188-aeb1-9a728654092f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc" Jul 22 12:18:51.136042 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:51.136017 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c909037b-917d-4188-aeb1-9a728654092f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-94d621ea-gshfc\" (UID: \"c909037b-917d-4188-aeb1-9a728654092f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc" Jul 22 12:18:51.310892 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:51.310804 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc" Jul 22 12:18:51.438964 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:51.438914 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc"] Jul 22 12:18:51.441518 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:18:51.441481 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc909037b_917d_4188_aeb1_9a728654092f.slice/crio-05248e6e31a34eea174fe1dc6b157fd6f64c4cd4d5a6738122fc671aaf16ade1 WatchSource:0}: Error finding container 05248e6e31a34eea174fe1dc6b157fd6f64c4cd4d5a6738122fc671aaf16ade1: Status 404 returned error can't find the container with id 05248e6e31a34eea174fe1dc6b157fd6f64c4cd4d5a6738122fc671aaf16ade1 Jul 22 12:18:52.107213 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:52.107177 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc" event={"ID":"c909037b-917d-4188-aeb1-9a728654092f","Type":"ContainerStarted","Data":"d4b628a3b46951f239f8f737bfd827d38f10ee1921d1a4134eed09ec06252156"} Jul 22 12:18:52.107593 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:52.107233 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc" event={"ID":"c909037b-917d-4188-aeb1-9a728654092f","Type":"ContainerStarted","Data":"05248e6e31a34eea174fe1dc6b157fd6f64c4cd4d5a6738122fc671aaf16ade1"} Jul 22 12:18:52.124692 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:52.124634 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc" podStartSLOduration=2.124618129 podStartE2EDuration="2.124618129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:18:52.123557506 +0000 UTC m=+1754.872769107" watchObservedRunningTime="2026-07-22 12:18:52.124618129 +0000 UTC m=+1754.873829730" Jul 22 12:18:53.113207 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:53.113174 2555 generic.go:356] "Generic (PLEG): container finished" podID="c909037b-917d-4188-aeb1-9a728654092f" containerID="d4b628a3b46951f239f8f737bfd827d38f10ee1921d1a4134eed09ec06252156" exitCode=0 Jul 22 12:18:53.113635 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:53.113266 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc" event={"ID":"c909037b-917d-4188-aeb1-9a728654092f","Type":"ContainerDied","Data":"d4b628a3b46951f239f8f737bfd827d38f10ee1921d1a4134eed09ec06252156"} Jul 22 12:18:54.243510 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:54.243484 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc" Jul 22 12:18:54.259649 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:54.259617 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c909037b-917d-4188-aeb1-9a728654092f-maas-api-service-ca\") pod \"c909037b-917d-4188-aeb1-9a728654092f\" (UID: \"c909037b-917d-4188-aeb1-9a728654092f\") " Jul 22 12:18:54.260015 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:54.259986 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c909037b-917d-4188-aeb1-9a728654092f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c909037b-917d-4188-aeb1-9a728654092f" (UID: "c909037b-917d-4188-aeb1-9a728654092f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:18:54.361102 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:54.361061 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c909037b-917d-4188-aeb1-9a728654092f-maas-api-service-ca\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:18:55.122699 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:55.122672 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc" Jul 22 12:18:55.122870 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:55.122672 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc" event={"ID":"c909037b-917d-4188-aeb1-9a728654092f","Type":"ContainerDied","Data":"05248e6e31a34eea174fe1dc6b157fd6f64c4cd4d5a6738122fc671aaf16ade1"} Jul 22 12:18:55.122870 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:18:55.122789 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05248e6e31a34eea174fe1dc6b157fd6f64c4cd4d5a6738122fc671aaf16ade1" Jul 22 12:19:01.544837 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:01.544804 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:04.573839 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:04.573800 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:19:06.078443 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:06.078403 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:06.078861 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:06.078656 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc"] Jul 22 12:19:06.081583 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:06.081563 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-94d621ea-gshfc"] Jul 22 12:19:06.237912 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:06.237867 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:07.958059 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:07.958026 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c909037b-917d-4188-aeb1-9a728654092f" path="/var/lib/kubelet/pods/c909037b-917d-4188-aeb1-9a728654092f/volumes" Jul 22 12:19:08.138350 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:08.138321 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:08.237756 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:08.237644 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:08.607393 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:08.607359 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:15.371639 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:15.371602 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:15.475823 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:15.475784 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:15.583292 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:15.583258 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:15.715588 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:15.715500 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:21.217595 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:21.217557 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:47.199298 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:47.199262 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:54.142599 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:54.142556 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:54.207753 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:54.207718 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:55.218215 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:55.218176 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:19:55.319185 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:19:55.319153 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:10.435124 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:10.435057 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:16.596196 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:16.596155 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:16.659716 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:16.659677 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:16.754643 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:16.754598 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:16.856586 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:16.856500 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:27.370816 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:27.370773 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:28.202580 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:28.202543 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:36.229033 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:36.229000 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:36.337973 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:36.337906 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:36.434830 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:36.434771 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:36.529232 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:36.529139 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:37.402754 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:37.402719 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:37.502647 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:37.502607 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:37.836674 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:37.836634 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:39.334011 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:39.333980 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:40.167397 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:40.167359 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:20:45.426959 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:45.426924 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:45.485155 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:45.485094 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:46.128746 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:46.128709 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:51.469742 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:51.469699 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:20:53.080832 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:53.080798 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:53.182426 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:53.182396 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:53.374301 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:53.374258 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:53.666911 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:53.666827 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:53.799347 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:53.799307 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:56.586371 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:56.586325 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:56.669141 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:56.669092 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:56.739966 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:56.739926 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:57.073732 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:57.073646 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:57.265859 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:57.265810 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:20:57.366652 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:20:57.366609 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:01.079097 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:01.079021 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:01.715517 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:01.715484 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:01.813083 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:01.813044 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:02.139160 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:02.139099 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:02.679831 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:02.679794 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:21:03.574326 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:03.574290 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:08.419590 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:08.419557 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:08.519848 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:08.519802 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:08.770391 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:08.770316 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7"] Jul 22 12:21:08.770635 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:08.770624 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c909037b-917d-4188-aeb1-9a728654092f" containerName="revoke-keys" Jul 22 12:21:08.770679 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:08.770636 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c909037b-917d-4188-aeb1-9a728654092f" containerName="revoke-keys" Jul 22 12:21:08.770757 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:08.770686 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c909037b-917d-4188-aeb1-9a728654092f" containerName="revoke-keys" Jul 22 12:21:08.773643 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:08.773619 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7" Jul 22 12:21:08.776019 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:08.775999 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kmhv\"" Jul 22 12:21:08.776126 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:08.776005 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 12:21:08.780714 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:08.780685 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7"] Jul 22 12:21:08.862174 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:08.862123 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42adc222-8511-426d-a482-edf0455d6549-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7\" (UID: \"42adc222-8511-426d-a482-edf0455d6549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7" Jul 22 12:21:08.963420 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:08.963366 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42adc222-8511-426d-a482-edf0455d6549-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7\" (UID: \"42adc222-8511-426d-a482-edf0455d6549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7" Jul 22 12:21:08.964044 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:08.964021 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42adc222-8511-426d-a482-edf0455d6549-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7\" (UID: \"42adc222-8511-426d-a482-edf0455d6549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7" Jul 22 12:21:09.084572 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:09.084544 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7" Jul 22 12:21:09.213997 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:09.213964 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7"] Jul 22 12:21:09.216913 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:21:09.216884 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42adc222_8511_426d_a482_edf0455d6549.slice/crio-3dc6642ffc559554f0e2379f9916594dbf0915dd79908c5b4311a9bbba14b711 WatchSource:0}: Error finding container 3dc6642ffc559554f0e2379f9916594dbf0915dd79908c5b4311a9bbba14b711: Status 404 returned error can't find the container with id 3dc6642ffc559554f0e2379f9916594dbf0915dd79908c5b4311a9bbba14b711 Jul 22 12:21:09.687331 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:09.687246 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7" event={"ID":"42adc222-8511-426d-a482-edf0455d6549","Type":"ContainerStarted","Data":"4ddfd530fa7c105285106f22f0be6e4d9ebaad2b1e52405ce11779dab70332f9"} Jul 22 12:21:09.687331 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:09.687300 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7" event={"ID":"42adc222-8511-426d-a482-edf0455d6549","Type":"ContainerStarted","Data":"3dc6642ffc559554f0e2379f9916594dbf0915dd79908c5b4311a9bbba14b711"} Jul 22 12:21:09.703547 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:09.703499 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7" podStartSLOduration=1.7034836530000002 podStartE2EDuration="1.703483653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:21:09.702487126 +0000 UTC m=+1892.451698728" watchObservedRunningTime="2026-07-22 12:21:09.703483653 +0000 UTC m=+1892.452695289" Jul 22 12:21:10.693680 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:10.693640 2555 generic.go:356] "Generic (PLEG): container finished" podID="42adc222-8511-426d-a482-edf0455d6549" containerID="4ddfd530fa7c105285106f22f0be6e4d9ebaad2b1e52405ce11779dab70332f9" exitCode=0 Jul 22 12:21:10.694082 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:10.693701 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7" event={"ID":"42adc222-8511-426d-a482-edf0455d6549","Type":"ContainerDied","Data":"4ddfd530fa7c105285106f22f0be6e4d9ebaad2b1e52405ce11779dab70332f9"} Jul 22 12:21:11.836977 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:11.836949 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7" Jul 22 12:21:11.890891 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:11.890843 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42adc222-8511-426d-a482-edf0455d6549-maas-api-service-ca\") pod \"42adc222-8511-426d-a482-edf0455d6549\" (UID: \"42adc222-8511-426d-a482-edf0455d6549\") " Jul 22 12:21:11.891246 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:11.891220 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42adc222-8511-426d-a482-edf0455d6549-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42adc222-8511-426d-a482-edf0455d6549" (UID: "42adc222-8511-426d-a482-edf0455d6549"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:21:11.992370 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:11.992333 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42adc222-8511-426d-a482-edf0455d6549-maas-api-service-ca\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:21:12.702846 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:12.702817 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7" Jul 22 12:21:12.703002 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:12.702849 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7" event={"ID":"42adc222-8511-426d-a482-edf0455d6549","Type":"ContainerDied","Data":"3dc6642ffc559554f0e2379f9916594dbf0915dd79908c5b4311a9bbba14b711"} Jul 22 12:21:12.703002 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:12.702890 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dc6642ffc559554f0e2379f9916594dbf0915dd79908c5b4311a9bbba14b711" Jul 22 12:21:19.215917 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:19.215878 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:22.063592 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:22.063554 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:21:23.846431 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:23.846379 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:23.850249 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:23.850223 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7"] Jul 22 12:21:23.852414 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:23.852392 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9b9afab-4pmb7"] Jul 22 12:21:23.959642 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:23.959598 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42adc222-8511-426d-a482-edf0455d6549" path="/var/lib/kubelet/pods/42adc222-8511-426d-a482-edf0455d6549/volumes" Jul 22 12:21:24.218056 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:24.217960 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:30.458059 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:30.458011 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:30.559588 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:30.559552 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:30.699128 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:30.699074 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:30.933899 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:30.933866 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:31.060665 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:31.060619 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:31.187693 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:31.187600 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:31.283435 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:31.283399 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:37.499734 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:37.499702 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:37.600182 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:37.600150 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:43.722655 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:43.722605 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:43.932009 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:43.931973 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:50.473099 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:50.473060 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:50.758012 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:50.757921 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:51.182073 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:51.182039 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:52.538393 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:52.538356 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:53.890322 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:53.890284 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:54.087062 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:54.087029 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:54.187352 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:54.187257 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:54.592142 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:54.592068 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:54.693928 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:54.693884 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:58.380898 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:58.380860 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:21:59.372566 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:21:59.372520 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:22:16.323731 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:16.323695 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:22:19.380779 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:19.380742 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:22:33.277040 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:33.276952 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:22:36.764985 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:36.764946 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4"] Jul 22 12:22:36.765419 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:36.765277 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42adc222-8511-426d-a482-edf0455d6549" containerName="revoke-keys" Jul 22 12:22:36.765419 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:36.765288 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42adc222-8511-426d-a482-edf0455d6549" containerName="revoke-keys" Jul 22 12:22:36.765419 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:36.765345 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42adc222-8511-426d-a482-edf0455d6549" containerName="revoke-keys" Jul 22 12:22:36.768499 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:36.768482 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4" Jul 22 12:22:36.770621 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:36.770599 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kmhv\"" Jul 22 12:22:36.770753 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:36.770651 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 12:22:36.775749 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:36.775719 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4"] Jul 22 12:22:36.824119 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:36.824084 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fbd64d-9459-4276-b03a-9fd3cc99ee62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4\" (UID: \"18fbd64d-9459-4276-b03a-9fd3cc99ee62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4" Jul 22 12:22:36.924675 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:36.924630 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fbd64d-9459-4276-b03a-9fd3cc99ee62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4\" (UID: \"18fbd64d-9459-4276-b03a-9fd3cc99ee62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4" Jul 22 12:22:36.925285 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:36.925267 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fbd64d-9459-4276-b03a-9fd3cc99ee62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4\" (UID: \"18fbd64d-9459-4276-b03a-9fd3cc99ee62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4" Jul 22 12:22:37.079369 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:37.079276 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4" Jul 22 12:22:37.213656 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:37.213559 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4"] Jul 22 12:22:37.216524 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:22:37.216492 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fbd64d_9459_4276_b03a_9fd3cc99ee62.slice/crio-02328883538b013d01929ba2b9e298ac5387eaa3fb438d05011195416b4996ba WatchSource:0}: Error finding container 02328883538b013d01929ba2b9e298ac5387eaa3fb438d05011195416b4996ba: Status 404 returned error can't find the container with id 02328883538b013d01929ba2b9e298ac5387eaa3fb438d05011195416b4996ba Jul 22 12:22:38.062510 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:38.062472 2555 generic.go:356] "Generic (PLEG): container finished" podID="18fbd64d-9459-4276-b03a-9fd3cc99ee62" containerID="1cf0177ef3c832b824f5646a41b8471f0ad014a819a4db96c329b4f1be197678" exitCode=0 Jul 22 12:22:38.062921 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:38.062540 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4" event={"ID":"18fbd64d-9459-4276-b03a-9fd3cc99ee62","Type":"ContainerDied","Data":"1cf0177ef3c832b824f5646a41b8471f0ad014a819a4db96c329b4f1be197678"} Jul 22 12:22:38.062921 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:38.062587 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4" event={"ID":"18fbd64d-9459-4276-b03a-9fd3cc99ee62","Type":"ContainerStarted","Data":"02328883538b013d01929ba2b9e298ac5387eaa3fb438d05011195416b4996ba"} Jul 22 12:22:39.197438 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:39.197413 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4" Jul 22 12:22:39.237633 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:39.237593 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fbd64d-9459-4276-b03a-9fd3cc99ee62-maas-api-service-ca\") pod \"18fbd64d-9459-4276-b03a-9fd3cc99ee62\" (UID: \"18fbd64d-9459-4276-b03a-9fd3cc99ee62\") " Jul 22 12:22:39.237985 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:39.237957 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18fbd64d-9459-4276-b03a-9fd3cc99ee62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18fbd64d-9459-4276-b03a-9fd3cc99ee62" (UID: "18fbd64d-9459-4276-b03a-9fd3cc99ee62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:22:39.338098 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:39.338064 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fbd64d-9459-4276-b03a-9fd3cc99ee62-maas-api-service-ca\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:22:40.073144 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:40.073102 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4" Jul 22 12:22:40.073144 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:40.073096 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4" event={"ID":"18fbd64d-9459-4276-b03a-9fd3cc99ee62","Type":"ContainerDied","Data":"02328883538b013d01929ba2b9e298ac5387eaa3fb438d05011195416b4996ba"} Jul 22 12:22:40.073355 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:40.073166 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02328883538b013d01929ba2b9e298ac5387eaa3fb438d05011195416b4996ba" Jul 22 12:22:50.676408 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:50.676370 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:22:52.227795 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:52.227761 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:22:56.870015 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:56.869979 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4"] Jul 22 12:22:56.873591 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:56.873563 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cd00702-g9gd4"] Jul 22 12:22:57.046022 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:57.045991 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:22:57.958358 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:57.958319 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18fbd64d-9459-4276-b03a-9fd3cc99ee62" path="/var/lib/kubelet/pods/18fbd64d-9459-4276-b03a-9fd3cc99ee62/volumes" Jul 22 12:22:59.101121 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:22:59.101084 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:01.374303 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:01.374267 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:23:04.183071 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:04.183038 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:20.305872 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:20.305836 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:25.125796 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:25.125744 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:25.222889 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:25.222860 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:40.504334 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:40.504297 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:41.646002 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:41.645965 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:48.127236 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:48.127201 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:48.227025 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:48.226992 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:48.407611 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:48.407515 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:48.637862 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:48.637827 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:49.776059 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:49.776026 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:49.880994 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:49.880957 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:50.712100 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:50.712064 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:50.840166 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:50.840129 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:50.985096 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:50.985006 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:52.386439 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:52.386400 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:23:52.462982 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:52.462949 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:59.008920 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:59.008874 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:59.113631 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:59.113595 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:23:59.897543 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:23:59.897503 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:02.692598 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:02.692563 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:24:07.074103 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:07.074061 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:07.184533 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:07.184495 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:07.322547 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:07.322506 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:17.859596 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:17.859553 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:18.230529 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:18.230438 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:37.823127 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:37.823075 2555 scope.go:117] "RemoveContainer" containerID="a644b8becfe130b550e1aafb989f16a0287d825f92d50a4e96395ee2bf81eb2a" Jul 22 12:24:39.142889 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:39.142854 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:45.620549 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:45.620508 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:45.720796 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:45.720763 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:45.980013 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:45.979920 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:46.308921 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:46.308830 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:46.426978 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:46.426946 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:48.930292 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:48.930255 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:49.029594 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:49.029557 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:49.173397 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:49.173360 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:49.565211 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:49.565176 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:49.669897 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:49.669865 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:49.771646 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:49.771602 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:53.435027 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:53.434984 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:53.786921 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:53.786835 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:54.103676 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:54.103645 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:54.204018 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:54.203966 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:54.673870 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:54.673840 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:24:56.936271 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:56.936229 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:57.052455 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:57.052423 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:57.177997 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:57.177956 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:57.585861 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:57.585824 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:57.686625 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:57.686589 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:24:57.786803 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:24:57.786767 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:25:01.473792 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:01.473753 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:25:07.129487 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:07.129450 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:25:10.577541 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:10.577414 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:25:10.678591 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:10.678561 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:25:10.973423 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:10.973381 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:25:18.481963 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:18.481927 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:25:18.580739 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:18.580699 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:25:21.483635 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:21.483598 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:25:26.767077 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:26.767033 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:25:36.198687 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.198603 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5"] Jul 22 12:25:36.199046 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.198939 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fbd64d-9459-4276-b03a-9fd3cc99ee62" containerName="revoke-keys" Jul 22 12:25:36.199046 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.198950 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fbd64d-9459-4276-b03a-9fd3cc99ee62" containerName="revoke-keys" Jul 22 12:25:36.199046 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.199001 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="18fbd64d-9459-4276-b03a-9fd3cc99ee62" containerName="revoke-keys" Jul 22 12:25:36.202074 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.202057 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.205489 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.205464 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ffe082\"/\"kube-root-ca.crt\"" Jul 22 12:25:36.205717 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.205701 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ffe082\"/\"default-dockercfg-xq5fb\"" Jul 22 12:25:36.205792 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.205734 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ffe082\"/\"openshift-service-ca.crt\"" Jul 22 12:25:36.205850 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.205817 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ffe082\"/\"test-model-ffe082-kserve-self-signed-certs\"" Jul 22 12:25:36.213220 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.213195 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5"] Jul 22 12:25:36.245012 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.244975 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:25:36.296168 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.296132 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-dshm\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.296364 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.296185 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-model-cache\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.296364 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.296278 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-home\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.296364 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.296308 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-tmp-dir\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.296364 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.296341 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jd2h\" (UniqueName: \"kubernetes.io/projected/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-kube-api-access-2jd2h\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.296513 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.296397 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-tls-certs\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.397168 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.397100 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-home\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.397168 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.397169 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-tmp-dir\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.397431 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.397193 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jd2h\" (UniqueName: \"kubernetes.io/projected/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-kube-api-access-2jd2h\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.397431 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.397220 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-tls-certs\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.397431 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.397338 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-dshm\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.397431 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.397395 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-model-cache\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.397683 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.397509 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-home\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.397683 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.397571 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-tmp-dir\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.397770 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.397710 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-model-cache\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.399584 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.399558 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-dshm\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.399784 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.399769 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-tls-certs\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.406455 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.406428 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jd2h\" (UniqueName: \"kubernetes.io/projected/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-kube-api-access-2jd2h\") pod \"test-model-ffe082-kserve-7bbc66d44c-9nfq5\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.511834 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.511739 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:36.645153 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.644996 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5"] Jul 22 12:25:36.647760 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:25:36.647730 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99aa80ce_6ac1_42c9_988a_371f2d73bbcb.slice/crio-b447d8b3b47bcc5e6f872d6a8b0c641fa32bae6f76770e25b2b57712bc5c72d7 WatchSource:0}: Error finding container b447d8b3b47bcc5e6f872d6a8b0c641fa32bae6f76770e25b2b57712bc5c72d7: Status 404 returned error can't find the container with id b447d8b3b47bcc5e6f872d6a8b0c641fa32bae6f76770e25b2b57712bc5c72d7 Jul 22 12:25:36.850814 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.850781 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" event={"ID":"99aa80ce-6ac1-42c9-988a-371f2d73bbcb","Type":"ContainerStarted","Data":"1005f921edf6dad62feada354c186d0a60800ef6ae8024ec1c1a8e7047521646"} Jul 22 12:25:36.851006 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.850828 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" event={"ID":"99aa80ce-6ac1-42c9-988a-371f2d73bbcb","Type":"ContainerStarted","Data":"b447d8b3b47bcc5e6f872d6a8b0c641fa32bae6f76770e25b2b57712bc5c72d7"} Jul 22 12:25:36.870305 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:36.870248 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" podStartSLOduration=0.870231943 podStartE2EDuration="870.231943ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:25:36.868848203 +0000 UTC m=+2159.618059805" watchObservedRunningTime="2026-07-22 12:25:36.870231943 +0000 UTC m=+2159.619443544" Jul 22 12:25:37.841076 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:37.841044 2555 scope.go:117] "RemoveContainer" containerID="d4b628a3b46951f239f8f737bfd827d38f10ee1921d1a4134eed09ec06252156" Jul 22 12:25:41.467012 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:41.466980 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:25:46.512048 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:46.511997 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:46.512048 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:46.512059 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:46.524465 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:46.524437 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:46.906206 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:46.906176 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:25:54.576552 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:25:54.576515 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:26:16.364895 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:16.364854 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:17.249170 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:17.249139 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5"] Jul 22 12:26:17.274230 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:17.274199 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:17.382657 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:17.382622 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh"] Jul 22 12:26:17.385810 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:17.385793 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh" Jul 22 12:26:17.388240 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:17.388217 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 12:26:17.388354 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:17.388224 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kmhv\"" Jul 22 12:26:17.394258 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:17.394234 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh"] Jul 22 12:26:17.552127 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:17.552019 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b680e417-9569-4c3d-9ea4-e037edb51132-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh\" (UID: \"b680e417-9569-4c3d-9ea4-e037edb51132\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh" Jul 22 12:26:17.653461 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:17.653427 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b680e417-9569-4c3d-9ea4-e037edb51132-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh\" (UID: \"b680e417-9569-4c3d-9ea4-e037edb51132\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh" Jul 22 12:26:17.654056 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:17.654035 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b680e417-9569-4c3d-9ea4-e037edb51132-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh\" (UID: \"b680e417-9569-4c3d-9ea4-e037edb51132\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh" Jul 22 12:26:17.696515 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:17.696482 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh" Jul 22 12:26:17.831700 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:17.831676 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh"] Jul 22 12:26:17.834988 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:26:17.834956 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb680e417_9569_4c3d_9ea4_e037edb51132.slice/crio-14d38c54cf3a7f56cf07a437f2051d410b3178ff690c7d5e9b3ca92de9e1383b WatchSource:0}: Error finding container 14d38c54cf3a7f56cf07a437f2051d410b3178ff690c7d5e9b3ca92de9e1383b: Status 404 returned error can't find the container with id 14d38c54cf3a7f56cf07a437f2051d410b3178ff690c7d5e9b3ca92de9e1383b Jul 22 12:26:17.913522 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:17.913485 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:18.011737 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:18.011701 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:18.023710 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:18.023679 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh" event={"ID":"b680e417-9569-4c3d-9ea4-e037edb51132","Type":"ContainerStarted","Data":"14d38c54cf3a7f56cf07a437f2051d410b3178ff690c7d5e9b3ca92de9e1383b"} Jul 22 12:26:19.030076 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:19.030038 2555 generic.go:356] "Generic (PLEG): container finished" podID="b680e417-9569-4c3d-9ea4-e037edb51132" containerID="34748cf845c1fcb3e76295b59c10ecf672bb02e9e829e645caede87738f5df99" exitCode=0 Jul 22 12:26:19.030469 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:19.030140 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh" event={"ID":"b680e417-9569-4c3d-9ea4-e037edb51132","Type":"ContainerDied","Data":"34748cf845c1fcb3e76295b59c10ecf672bb02e9e829e645caede87738f5df99"} Jul 22 12:26:20.164095 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:20.164068 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh" Jul 22 12:26:20.278932 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:20.278895 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b680e417-9569-4c3d-9ea4-e037edb51132-maas-api-service-ca\") pod \"b680e417-9569-4c3d-9ea4-e037edb51132\" (UID: \"b680e417-9569-4c3d-9ea4-e037edb51132\") " Jul 22 12:26:20.279305 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:20.279281 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b680e417-9569-4c3d-9ea4-e037edb51132-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b680e417-9569-4c3d-9ea4-e037edb51132" (UID: "b680e417-9569-4c3d-9ea4-e037edb51132"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:26:20.379828 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:20.379788 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b680e417-9569-4c3d-9ea4-e037edb51132-maas-api-service-ca\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:26:20.677859 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:20.677770 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:26:21.040275 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:21.040195 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh" Jul 22 12:26:21.040414 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:21.040189 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh" event={"ID":"b680e417-9569-4c3d-9ea4-e037edb51132","Type":"ContainerDied","Data":"14d38c54cf3a7f56cf07a437f2051d410b3178ff690c7d5e9b3ca92de9e1383b"} Jul 22 12:26:21.040414 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:21.040320 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14d38c54cf3a7f56cf07a437f2051d410b3178ff690c7d5e9b3ca92de9e1383b" Jul 22 12:26:27.834967 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:27.834904 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:32.269308 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.269270 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" podUID="99aa80ce-6ac1-42c9-988a-371f2d73bbcb" containerName="main" containerID="cri-o://1005f921edf6dad62feada354c186d0a60800ef6ae8024ec1c1a8e7047521646" gracePeriod=45 Jul 22 12:26:32.462659 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.462625 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:32.474325 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.474297 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh"] Jul 22 12:26:32.477581 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.477557 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-53f570-rx9nh"] Jul 22 12:26:32.522446 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.522386 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:26:32.577535 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.577498 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jd2h\" (UniqueName: \"kubernetes.io/projected/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-kube-api-access-2jd2h\") pod \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " Jul 22 12:26:32.577535 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.577543 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-dshm\") pod \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " Jul 22 12:26:32.577754 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.577588 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-tmp-dir\") pod \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " Jul 22 12:26:32.577754 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.577685 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-tls-certs\") pod \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " Jul 22 12:26:32.577754 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.577729 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-home\") pod \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " Jul 22 12:26:32.577915 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.577771 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-model-cache\") pod \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\" (UID: \"99aa80ce-6ac1-42c9-988a-371f2d73bbcb\") " Jul 22 12:26:32.577915 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.577834 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99aa80ce-6ac1-42c9-988a-371f2d73bbcb" (UID: "99aa80ce-6ac1-42c9-988a-371f2d73bbcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:26:32.578073 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.578042 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "99aa80ce-6ac1-42c9-988a-371f2d73bbcb" (UID: "99aa80ce-6ac1-42c9-988a-371f2d73bbcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:26:32.578073 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.578053 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-home" (OuterVolumeSpecName: "home") pod "99aa80ce-6ac1-42c9-988a-371f2d73bbcb" (UID: "99aa80ce-6ac1-42c9-988a-371f2d73bbcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:26:32.578252 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.578101 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-tmp-dir\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:26:32.579856 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.579829 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99aa80ce-6ac1-42c9-988a-371f2d73bbcb" (UID: "99aa80ce-6ac1-42c9-988a-371f2d73bbcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:26:32.579951 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.579851 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-kube-api-access-2jd2h" (OuterVolumeSpecName: "kube-api-access-2jd2h") pod "99aa80ce-6ac1-42c9-988a-371f2d73bbcb" (UID: "99aa80ce-6ac1-42c9-988a-371f2d73bbcb"). InnerVolumeSpecName "kube-api-access-2jd2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:26:32.579951 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.579892 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-dshm" (OuterVolumeSpecName: "dshm") pod "99aa80ce-6ac1-42c9-988a-371f2d73bbcb" (UID: "99aa80ce-6ac1-42c9-988a-371f2d73bbcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:26:32.678665 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.678633 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-tls-certs\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:26:32.678665 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.678661 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-home\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:26:32.678665 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.678670 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-model-cache\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:26:32.678887 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.678679 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jd2h\" (UniqueName: \"kubernetes.io/projected/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-kube-api-access-2jd2h\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:26:32.678887 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.678687 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99aa80ce-6ac1-42c9-988a-371f2d73bbcb-dshm\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:26:32.844764 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:32.844706 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:33.092050 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:33.092017 2555 generic.go:356] "Generic (PLEG): container finished" podID="99aa80ce-6ac1-42c9-988a-371f2d73bbcb" containerID="1005f921edf6dad62feada354c186d0a60800ef6ae8024ec1c1a8e7047521646" exitCode=0 Jul 22 12:26:33.092278 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:33.092066 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" Jul 22 12:26:33.092278 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:33.092124 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" event={"ID":"99aa80ce-6ac1-42c9-988a-371f2d73bbcb","Type":"ContainerDied","Data":"1005f921edf6dad62feada354c186d0a60800ef6ae8024ec1c1a8e7047521646"} Jul 22 12:26:33.092278 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:33.092159 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5" event={"ID":"99aa80ce-6ac1-42c9-988a-371f2d73bbcb","Type":"ContainerDied","Data":"b447d8b3b47bcc5e6f872d6a8b0c641fa32bae6f76770e25b2b57712bc5c72d7"} Jul 22 12:26:33.092278 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:33.092179 2555 scope.go:117] "RemoveContainer" containerID="1005f921edf6dad62feada354c186d0a60800ef6ae8024ec1c1a8e7047521646" Jul 22 12:26:33.100592 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:33.100574 2555 scope.go:117] "RemoveContainer" containerID="1005f921edf6dad62feada354c186d0a60800ef6ae8024ec1c1a8e7047521646" Jul 22 12:26:33.100837 ip-10-0-130-190 kubenswrapper[2555]: E0722 12:26:33.100816 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1005f921edf6dad62feada354c186d0a60800ef6ae8024ec1c1a8e7047521646\": container with ID starting with 1005f921edf6dad62feada354c186d0a60800ef6ae8024ec1c1a8e7047521646 not found: ID does not exist" containerID="1005f921edf6dad62feada354c186d0a60800ef6ae8024ec1c1a8e7047521646" Jul 22 12:26:33.100894 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:33.100844 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1005f921edf6dad62feada354c186d0a60800ef6ae8024ec1c1a8e7047521646"} err="failed to get container status \"1005f921edf6dad62feada354c186d0a60800ef6ae8024ec1c1a8e7047521646\": rpc error: code = NotFound desc = could not find container \"1005f921edf6dad62feada354c186d0a60800ef6ae8024ec1c1a8e7047521646\": container with ID starting with 1005f921edf6dad62feada354c186d0a60800ef6ae8024ec1c1a8e7047521646 not found: ID does not exist" Jul 22 12:26:33.113560 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:33.113525 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5"] Jul 22 12:26:33.118122 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:33.118086 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ffe082/test-model-ffe082-kserve-7bbc66d44c-9nfq5"] Jul 22 12:26:33.364659 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:33.364576 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:26:33.958957 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:33.958923 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99aa80ce-6ac1-42c9-988a-371f2d73bbcb" path="/var/lib/kubelet/pods/99aa80ce-6ac1-42c9-988a-371f2d73bbcb/volumes" Jul 22 12:26:33.959267 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:33.959254 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b680e417-9569-4c3d-9ea4-e037edb51132" path="/var/lib/kubelet/pods/b680e417-9569-4c3d-9ea4-e037edb51132/volumes" Jul 22 12:26:39.446257 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:39.446212 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:39.512206 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:39.512166 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:39.743861 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:39.743768 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:39.958012 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:39.957975 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:40.069885 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.069808 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn"] Jul 22 12:26:40.070135 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.070121 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99aa80ce-6ac1-42c9-988a-371f2d73bbcb" containerName="main" Jul 22 12:26:40.070182 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.070138 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="99aa80ce-6ac1-42c9-988a-371f2d73bbcb" containerName="main" Jul 22 12:26:40.070182 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.070156 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b680e417-9569-4c3d-9ea4-e037edb51132" containerName="revoke-keys" Jul 22 12:26:40.070182 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.070161 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b680e417-9569-4c3d-9ea4-e037edb51132" containerName="revoke-keys" Jul 22 12:26:40.070273 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.070213 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="99aa80ce-6ac1-42c9-988a-371f2d73bbcb" containerName="main" Jul 22 12:26:40.070273 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.070223 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b680e417-9569-4c3d-9ea4-e037edb51132" containerName="revoke-keys" Jul 22 12:26:40.074637 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.074620 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn" Jul 22 12:26:40.076919 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.076898 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 12:26:40.077030 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.076924 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kmhv\"" Jul 22 12:26:40.083805 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.083778 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn"] Jul 22 12:26:40.136533 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.136500 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743f9ce9-41e3-4724-a765-4d87dcc7b21e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn\" (UID: \"743f9ce9-41e3-4724-a765-4d87dcc7b21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn" Jul 22 12:26:40.237060 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.237022 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743f9ce9-41e3-4724-a765-4d87dcc7b21e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn\" (UID: \"743f9ce9-41e3-4724-a765-4d87dcc7b21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn" Jul 22 12:26:40.237655 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.237638 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743f9ce9-41e3-4724-a765-4d87dcc7b21e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn\" (UID: \"743f9ce9-41e3-4724-a765-4d87dcc7b21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn" Jul 22 12:26:40.385408 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.385374 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn" Jul 22 12:26:40.519023 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.519002 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn"] Jul 22 12:26:40.521231 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:26:40.521203 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743f9ce9_41e3_4724_a765_4d87dcc7b21e.slice/crio-68255a9fd3b9815ea6a76696b5ab8f84efb5b79960737666c180aab7eb743366 WatchSource:0}: Error finding container 68255a9fd3b9815ea6a76696b5ab8f84efb5b79960737666c180aab7eb743366: Status 404 returned error can't find the container with id 68255a9fd3b9815ea6a76696b5ab8f84efb5b79960737666c180aab7eb743366 Jul 22 12:26:40.697701 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.697670 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:40.798250 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:40.798208 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:41.129245 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:41.129214 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn" event={"ID":"743f9ce9-41e3-4724-a765-4d87dcc7b21e","Type":"ContainerStarted","Data":"4f27f2fde1e8fb4fdaa84e40d79572baabed80cc37b3200d3a086414a98a3772"} Jul 22 12:26:41.129438 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:41.129264 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn" event={"ID":"743f9ce9-41e3-4724-a765-4d87dcc7b21e","Type":"ContainerStarted","Data":"68255a9fd3b9815ea6a76696b5ab8f84efb5b79960737666c180aab7eb743366"} Jul 22 12:26:41.146963 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:41.146912 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn" podStartSLOduration=1.146895739 podStartE2EDuration="1.146895739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:26:41.145769816 +0000 UTC m=+2223.894981419" watchObservedRunningTime="2026-07-22 12:26:41.146895739 +0000 UTC m=+2223.896107341" Jul 22 12:26:42.135155 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:42.135119 2555 generic.go:356] "Generic (PLEG): container finished" podID="743f9ce9-41e3-4724-a765-4d87dcc7b21e" containerID="4f27f2fde1e8fb4fdaa84e40d79572baabed80cc37b3200d3a086414a98a3772" exitCode=0 Jul 22 12:26:42.135525 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:42.135131 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn" event={"ID":"743f9ce9-41e3-4724-a765-4d87dcc7b21e","Type":"ContainerDied","Data":"4f27f2fde1e8fb4fdaa84e40d79572baabed80cc37b3200d3a086414a98a3772"} Jul 22 12:26:43.268302 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:43.268276 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn" Jul 22 12:26:43.363922 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:43.363891 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743f9ce9-41e3-4724-a765-4d87dcc7b21e-maas-api-service-ca\") pod \"743f9ce9-41e3-4724-a765-4d87dcc7b21e\" (UID: \"743f9ce9-41e3-4724-a765-4d87dcc7b21e\") " Jul 22 12:26:43.364293 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:43.364270 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743f9ce9-41e3-4724-a765-4d87dcc7b21e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "743f9ce9-41e3-4724-a765-4d87dcc7b21e" (UID: "743f9ce9-41e3-4724-a765-4d87dcc7b21e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:26:43.464518 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:43.464430 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743f9ce9-41e3-4724-a765-4d87dcc7b21e-maas-api-service-ca\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:26:44.145991 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:44.145963 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn" Jul 22 12:26:44.146179 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:44.145965 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn" event={"ID":"743f9ce9-41e3-4724-a765-4d87dcc7b21e","Type":"ContainerDied","Data":"68255a9fd3b9815ea6a76696b5ab8f84efb5b79960737666c180aab7eb743366"} Jul 22 12:26:44.146179 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:44.146078 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68255a9fd3b9815ea6a76696b5ab8f84efb5b79960737666c180aab7eb743366" Jul 22 12:26:50.530630 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:50.530585 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:51.802957 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:51.802926 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:53.465527 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:53.465489 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:26:55.150747 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:55.150714 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn"] Jul 22 12:26:55.154022 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:55.153995 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe082-nmxdn"] Jul 22 12:26:55.154816 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:55.154788 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:55.958585 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:55.958547 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="743f9ce9-41e3-4724-a765-4d87dcc7b21e" path="/var/lib/kubelet/pods/743f9ce9-41e3-4724-a765-4d87dcc7b21e/volumes" Jul 22 12:26:56.960999 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:56.960964 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:26:59.962550 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:26:59.962455 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:02.095594 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:02.095558 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:02.201577 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:02.201537 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:02.468709 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:02.468607 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:02.611914 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:02.611874 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:02.951479 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:02.951444 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:03.280520 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:03.280434 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:03.380246 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:03.380211 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:06.835579 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:06.835541 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:06.929180 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:06.929145 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:07.029447 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:07.029381 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:07.127660 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:07.127573 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:07.432437 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:07.432351 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:07.532892 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:07.532848 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:11.319811 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:11.319762 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:11.690882 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:11.690843 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:12.196249 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:12.196212 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:12.392785 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:12.392748 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:12.570309 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:12.570222 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:27:14.728498 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:14.728453 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:14.812266 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:14.812220 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:14.891704 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:14.891665 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:15.417000 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:15.416964 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:15.518014 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:15.517977 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:19.229023 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:19.228988 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:27.256837 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.256784 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj"] Jul 22 12:27:27.257279 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.257090 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="743f9ce9-41e3-4724-a765-4d87dcc7b21e" containerName="revoke-keys" Jul 22 12:27:27.257279 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.257101 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="743f9ce9-41e3-4724-a765-4d87dcc7b21e" containerName="revoke-keys" Jul 22 12:27:27.257279 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.257173 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="743f9ce9-41e3-4724-a765-4d87dcc7b21e" containerName="revoke-keys" Jul 22 12:27:27.260156 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.260139 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj" Jul 22 12:27:27.262620 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.262596 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 12:27:27.262728 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.262678 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kmhv\"" Jul 22 12:27:27.268322 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.268298 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj"] Jul 22 12:27:27.346274 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.346241 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab6193d-ac99-4e65-aa7a-3539f4c81442-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e96939-k9nvj\" (UID: \"7ab6193d-ac99-4e65-aa7a-3539f4c81442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj" Jul 22 12:27:27.447619 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.447577 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab6193d-ac99-4e65-aa7a-3539f4c81442-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e96939-k9nvj\" (UID: \"7ab6193d-ac99-4e65-aa7a-3539f4c81442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj" Jul 22 12:27:27.448312 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.448291 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab6193d-ac99-4e65-aa7a-3539f4c81442-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e96939-k9nvj\" (UID: \"7ab6193d-ac99-4e65-aa7a-3539f4c81442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj" Jul 22 12:27:27.570889 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.570782 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj" Jul 22 12:27:27.709412 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.709377 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj"] Jul 22 12:27:27.713787 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:27:27.713756 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab6193d_ac99_4e65_aa7a_3539f4c81442.slice/crio-24b429df65080dcd7379814026eb770d67bb239147f353ce800bc35629d55d57 WatchSource:0}: Error finding container 24b429df65080dcd7379814026eb770d67bb239147f353ce800bc35629d55d57: Status 404 returned error can't find the container with id 24b429df65080dcd7379814026eb770d67bb239147f353ce800bc35629d55d57 Jul 22 12:27:27.792387 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.792344 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:27.891450 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:27.891402 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:28.334417 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:28.334381 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj" event={"ID":"7ab6193d-ac99-4e65-aa7a-3539f4c81442","Type":"ContainerStarted","Data":"0a3b6898f1f89f13d9559d5dd58c7e8b7508083c08e4efeeb60f709cfe45b3b6"} Jul 22 12:27:28.334800 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:28.334431 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj" event={"ID":"7ab6193d-ac99-4e65-aa7a-3539f4c81442","Type":"ContainerStarted","Data":"24b429df65080dcd7379814026eb770d67bb239147f353ce800bc35629d55d57"} Jul 22 12:27:28.351354 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:28.351306 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj" podStartSLOduration=1.351290602 podStartE2EDuration="1.351290602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:27:28.350007847 +0000 UTC m=+2271.099219450" watchObservedRunningTime="2026-07-22 12:27:28.351290602 +0000 UTC m=+2271.100502204" Jul 22 12:27:29.339746 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:29.339708 2555 generic.go:356] "Generic (PLEG): container finished" podID="7ab6193d-ac99-4e65-aa7a-3539f4c81442" containerID="0a3b6898f1f89f13d9559d5dd58c7e8b7508083c08e4efeeb60f709cfe45b3b6" exitCode=0 Jul 22 12:27:29.340174 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:29.339785 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj" event={"ID":"7ab6193d-ac99-4e65-aa7a-3539f4c81442","Type":"ContainerDied","Data":"0a3b6898f1f89f13d9559d5dd58c7e8b7508083c08e4efeeb60f709cfe45b3b6"} Jul 22 12:27:29.372869 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:29.372835 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:27:30.480537 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:30.480511 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj" Jul 22 12:27:30.575474 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:30.575445 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab6193d-ac99-4e65-aa7a-3539f4c81442-maas-api-service-ca\") pod \"7ab6193d-ac99-4e65-aa7a-3539f4c81442\" (UID: \"7ab6193d-ac99-4e65-aa7a-3539f4c81442\") " Jul 22 12:27:30.575815 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:30.575792 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab6193d-ac99-4e65-aa7a-3539f4c81442-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ab6193d-ac99-4e65-aa7a-3539f4c81442" (UID: "7ab6193d-ac99-4e65-aa7a-3539f4c81442"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:27:30.676833 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:30.676750 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab6193d-ac99-4e65-aa7a-3539f4c81442-maas-api-service-ca\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:27:31.349851 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:31.349819 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj" Jul 22 12:27:31.350031 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:31.349849 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj" event={"ID":"7ab6193d-ac99-4e65-aa7a-3539f4c81442","Type":"ContainerDied","Data":"24b429df65080dcd7379814026eb770d67bb239147f353ce800bc35629d55d57"} Jul 22 12:27:31.350031 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:31.349895 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24b429df65080dcd7379814026eb770d67bb239147f353ce800bc35629d55d57" Jul 22 12:27:36.206121 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:36.206070 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:36.306764 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:36.306728 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:37.868572 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:37.868534 2555 scope.go:117] "RemoveContainer" containerID="4ddfd530fa7c105285106f22f0be6e4d9ebaad2b1e52405ce11779dab70332f9" Jul 22 12:27:37.908422 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:37.908391 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:38.012048 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:38.012014 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:43.181735 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:43.181703 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:47.343564 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:47.343532 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj"] Jul 22 12:27:47.344011 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:47.343908 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:47.346277 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:47.346254 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e96939-k9nvj"] Jul 22 12:27:47.745037 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:47.744950 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:47.768262 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:47.768218 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:27:47.959246 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:47.959208 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ab6193d-ac99-4e65-aa7a-3539f4c81442" path="/var/lib/kubelet/pods/7ab6193d-ac99-4e65-aa7a-3539f4c81442/volumes" Jul 22 12:27:54.361252 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:54.361212 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:54.443517 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:54.443480 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:54.608883 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:54.608839 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:54.832764 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:54.832733 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:54.956717 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:54.956686 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8"] Jul 22 12:27:54.957005 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:54.956992 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab6193d-ac99-4e65-aa7a-3539f4c81442" containerName="revoke-keys" Jul 22 12:27:54.957005 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:54.957006 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab6193d-ac99-4e65-aa7a-3539f4c81442" containerName="revoke-keys" Jul 22 12:27:54.957133 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:54.957056 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab6193d-ac99-4e65-aa7a-3539f4c81442" containerName="revoke-keys" Jul 22 12:27:54.960146 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:54.960120 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8" Jul 22 12:27:54.966371 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:54.966349 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kmhv\"" Jul 22 12:27:54.966629 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:54.966616 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 12:27:54.975351 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:54.975322 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8"] Jul 22 12:27:55.083733 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:55.083650 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91fcf50d-1b0e-411c-bf43-96b1fef7c0d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8\" (UID: \"91fcf50d-1b0e-411c-bf43-96b1fef7c0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8" Jul 22 12:27:55.184597 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:55.184549 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91fcf50d-1b0e-411c-bf43-96b1fef7c0d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8\" (UID: \"91fcf50d-1b0e-411c-bf43-96b1fef7c0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8" Jul 22 12:27:55.185259 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:55.185238 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91fcf50d-1b0e-411c-bf43-96b1fef7c0d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8\" (UID: \"91fcf50d-1b0e-411c-bf43-96b1fef7c0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8" Jul 22 12:27:55.270668 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:55.270630 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8" Jul 22 12:27:55.403580 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:55.403549 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8"] Jul 22 12:27:55.407042 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:27:55.407012 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fcf50d_1b0e_411c_bf43_96b1fef7c0d5.slice/crio-9fdf0230532b69bfb7fc691fdcfafb748398e102e612448b2d3226ceacef3bc0 WatchSource:0}: Error finding container 9fdf0230532b69bfb7fc691fdcfafb748398e102e612448b2d3226ceacef3bc0: Status 404 returned error can't find the container with id 9fdf0230532b69bfb7fc691fdcfafb748398e102e612448b2d3226ceacef3bc0 Jul 22 12:27:55.453155 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:55.453125 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8" event={"ID":"91fcf50d-1b0e-411c-bf43-96b1fef7c0d5","Type":"ContainerStarted","Data":"9fdf0230532b69bfb7fc691fdcfafb748398e102e612448b2d3226ceacef3bc0"} Jul 22 12:27:55.494221 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:55.494187 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:55.591619 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:55.591583 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:27:56.459193 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:56.459077 2555 generic.go:356] "Generic (PLEG): container finished" podID="91fcf50d-1b0e-411c-bf43-96b1fef7c0d5" containerID="899a0dd2c27be05c02ac2a4488c6cf0a8a8c76ad05c95dd0916be96b9ee2447a" exitCode=0 Jul 22 12:27:56.459193 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:56.459149 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8" event={"ID":"91fcf50d-1b0e-411c-bf43-96b1fef7c0d5","Type":"ContainerDied","Data":"899a0dd2c27be05c02ac2a4488c6cf0a8a8c76ad05c95dd0916be96b9ee2447a"} Jul 22 12:27:57.589658 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:57.589634 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8" Jul 22 12:27:57.706405 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:57.706367 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91fcf50d-1b0e-411c-bf43-96b1fef7c0d5-maas-api-service-ca\") pod \"91fcf50d-1b0e-411c-bf43-96b1fef7c0d5\" (UID: \"91fcf50d-1b0e-411c-bf43-96b1fef7c0d5\") " Jul 22 12:27:57.706734 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:57.706710 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fcf50d-1b0e-411c-bf43-96b1fef7c0d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91fcf50d-1b0e-411c-bf43-96b1fef7c0d5" (UID: "91fcf50d-1b0e-411c-bf43-96b1fef7c0d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:27:57.807863 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:57.807769 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91fcf50d-1b0e-411c-bf43-96b1fef7c0d5-maas-api-service-ca\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:27:58.468630 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:58.468600 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8" Jul 22 12:27:58.468630 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:58.468614 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8" event={"ID":"91fcf50d-1b0e-411c-bf43-96b1fef7c0d5","Type":"ContainerDied","Data":"9fdf0230532b69bfb7fc691fdcfafb748398e102e612448b2d3226ceacef3bc0"} Jul 22 12:27:58.468831 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:27:58.468671 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fdf0230532b69bfb7fc691fdcfafb748398e102e612448b2d3226ceacef3bc0" Jul 22 12:28:10.432922 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:10.432879 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:14.971054 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:14.971020 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:28:15.043893 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:15.043856 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8"] Jul 22 12:28:15.047872 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:15.047840 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9edc58-qfkl8"] Jul 22 12:28:15.048753 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:15.048722 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:15.457070 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:15.457018 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:15.958443 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:15.958410 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91fcf50d-1b0e-411c-bf43-96b1fef7c0d5" path="/var/lib/kubelet/pods/91fcf50d-1b0e-411c-bf43-96b1fef7c0d5/volumes" Jul 22 12:28:21.817988 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:21.817943 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:21.876603 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:21.876569 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:22.073757 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:22.073664 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:22.309740 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:22.309699 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:22.643948 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:22.643910 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:22.966116 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:22.966005 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:23.175171 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:23.175135 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:25.988041 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:25.988008 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:26.090947 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:26.090910 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:26.251980 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:26.251898 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:26.380382 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:26.380352 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:26.489015 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:26.488986 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:26.600936 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:26.600903 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:26.913031 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:26.912942 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:27.011155 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:27.011093 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:30.757684 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:30.757570 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:31.121712 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:31.121679 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:31.436165 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:31.436054 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:31.715923 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:31.715834 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:32.371739 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:32.371704 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:28:34.271345 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:34.271299 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:34.363818 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:34.363784 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:34.450135 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:34.450083 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:34.901030 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:34.900996 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:34.999452 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:34.999413 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:37.887760 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:37.887729 2555 scope.go:117] "RemoveContainer" containerID="1cf0177ef3c832b824f5646a41b8471f0ad014a819a4db96c329b4f1be197678" Jul 22 12:28:38.793887 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:38.793848 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:47.711311 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:47.711274 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:47.814732 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:47.814697 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:48.368785 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:48.368748 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:28:52.531166 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:52.531130 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:54.592976 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:54.592935 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:54.652219 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:54.652178 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:55.668616 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:55.668585 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:55.768978 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:55.768933 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:28:57.475073 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:28:57.475036 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:29:28.748208 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:29:28.748168 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:29:28.847195 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:29:28.847160 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:29:32.581607 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:29:32.581570 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:29:46.286692 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:29:46.286651 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:30:00.141467 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.138136 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62"] Jul 22 12:30:00.141467 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.138879 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91fcf50d-1b0e-411c-bf43-96b1fef7c0d5" containerName="revoke-keys" Jul 22 12:30:00.141467 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.138901 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="91fcf50d-1b0e-411c-bf43-96b1fef7c0d5" containerName="revoke-keys" Jul 22 12:30:00.141467 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.139044 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="91fcf50d-1b0e-411c-bf43-96b1fef7c0d5" containerName="revoke-keys" Jul 22 12:30:00.143524 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.143489 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62" Jul 22 12:30:00.145914 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.145891 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745390-mlfdw"] Jul 22 12:30:00.146194 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.146173 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kmhv\"" Jul 22 12:30:00.149044 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.149021 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb"] Jul 22 12:30:00.149282 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.149265 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745390-mlfdw" Jul 22 12:30:00.152405 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.152383 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62"] Jul 22 12:30:00.152585 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.152569 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745390-mlfdw"] Jul 22 12:30:00.152712 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.152508 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb" Jul 22 12:30:00.155552 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.155528 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb"] Jul 22 12:30:00.160170 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.160147 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62" Jul 22 12:30:00.169925 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.169902 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745390-mlfdw" Jul 22 12:30:00.199287 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.199262 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb" Jul 22 12:30:00.317918 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.317421 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62"] Jul 22 12:30:00.371745 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.371722 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb"] Jul 22 12:30:00.374282 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:30:00.374249 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20114a93_6b95_4619_bfb3_a386a6ffba57.slice/crio-2b5ff735739f5486f2aa2b2cf25068bcfce3c27932cdd57213ff8ffa6c4d37a3 WatchSource:0}: Error finding container 2b5ff735739f5486f2aa2b2cf25068bcfce3c27932cdd57213ff8ffa6c4d37a3: Status 404 returned error can't find the container with id 2b5ff735739f5486f2aa2b2cf25068bcfce3c27932cdd57213ff8ffa6c4d37a3 Jul 22 12:30:00.546050 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.546006 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745390-mlfdw"] Jul 22 12:30:00.548665 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:30:00.548635 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f084fd_2c7b_4da3_b095_9db4cbf1e06b.slice/crio-65e882afd2e118844acb262b5acd61269afb807b22b1462204c228285f8a9925 WatchSource:0}: Error finding container 65e882afd2e118844acb262b5acd61269afb807b22b1462204c228285f8a9925: Status 404 returned error can't find the container with id 65e882afd2e118844acb262b5acd61269afb807b22b1462204c228285f8a9925 Jul 22 12:30:00.999939 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.999904 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb" event={"ID":"20114a93-6b95-4619-bfb3-a386a6ffba57","Type":"ContainerStarted","Data":"37542ca596d12924ef9a628360ad21460ce977871a58377291432db364591069"} Jul 22 12:30:01.000176 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:00.999961 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb" event={"ID":"20114a93-6b95-4619-bfb3-a386a6ffba57","Type":"ContainerStarted","Data":"2b5ff735739f5486f2aa2b2cf25068bcfce3c27932cdd57213ff8ffa6c4d37a3"} Jul 22 12:30:01.001442 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:01.001389 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745390-mlfdw" event={"ID":"18f084fd-2c7b-4da3-b095-9db4cbf1e06b","Type":"ContainerStarted","Data":"fb5c0bad7616a43b1ec00bb0ec53a70583a97d5198cae93aede836afbeaacf46"} Jul 22 12:30:01.001517 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:01.001468 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745390-mlfdw" event={"ID":"18f084fd-2c7b-4da3-b095-9db4cbf1e06b","Type":"ContainerStarted","Data":"65e882afd2e118844acb262b5acd61269afb807b22b1462204c228285f8a9925"} Jul 22 12:30:01.002771 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:01.002745 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62" event={"ID":"a83064a4-e9ba-43b7-a3aa-5ed9fb01bd1f","Type":"ContainerStarted","Data":"5478b64bf6f27a15b9bad896e500f32cb1e75b0f05d4a500fad29499380784fe"} Jul 22 12:30:01.002881 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:01.002784 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62" event={"ID":"a83064a4-e9ba-43b7-a3aa-5ed9fb01bd1f","Type":"ContainerStarted","Data":"cdbecb325ca5c8930b3f12b6f2ffda225fb844005dc759a74c01e2b4c5666fd3"} Jul 22 12:30:01.017530 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:01.017413 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb" podStartSLOduration=1.017391768 podStartE2EDuration="1.017391768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:30:01.016915709 +0000 UTC m=+2423.766127313" watchObservedRunningTime="2026-07-22 12:30:01.017391768 +0000 UTC m=+2423.766603371" Jul 22 12:30:01.033828 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:01.033772 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62" podStartSLOduration=1.033756649 podStartE2EDuration="1.033756649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:30:01.032738103 +0000 UTC m=+2423.781949706" watchObservedRunningTime="2026-07-22 12:30:01.033756649 +0000 UTC m=+2423.782968251" Jul 22 12:30:01.050923 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:01.050869 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745390-mlfdw" podStartSLOduration=1.050854557 podStartE2EDuration="1.050854557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:30:01.049577347 +0000 UTC m=+2423.798788952" watchObservedRunningTime="2026-07-22 12:30:01.050854557 +0000 UTC m=+2423.800066158" Jul 22 12:30:02.009097 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:02.009057 2555 generic.go:356] "Generic (PLEG): container finished" podID="a83064a4-e9ba-43b7-a3aa-5ed9fb01bd1f" containerID="5478b64bf6f27a15b9bad896e500f32cb1e75b0f05d4a500fad29499380784fe" exitCode=0 Jul 22 12:30:02.009586 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:02.009140 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62" event={"ID":"a83064a4-e9ba-43b7-a3aa-5ed9fb01bd1f","Type":"ContainerDied","Data":"5478b64bf6f27a15b9bad896e500f32cb1e75b0f05d4a500fad29499380784fe"} Jul 22 12:30:02.010638 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:02.010613 2555 generic.go:356] "Generic (PLEG): container finished" podID="20114a93-6b95-4619-bfb3-a386a6ffba57" containerID="37542ca596d12924ef9a628360ad21460ce977871a58377291432db364591069" exitCode=0 Jul 22 12:30:02.010772 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:02.010707 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb" event={"ID":"20114a93-6b95-4619-bfb3-a386a6ffba57","Type":"ContainerDied","Data":"37542ca596d12924ef9a628360ad21460ce977871a58377291432db364591069"} Jul 22 12:30:02.011993 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:02.011976 2555 generic.go:356] "Generic (PLEG): container finished" podID="18f084fd-2c7b-4da3-b095-9db4cbf1e06b" containerID="fb5c0bad7616a43b1ec00bb0ec53a70583a97d5198cae93aede836afbeaacf46" exitCode=0 Jul 22 12:30:02.012054 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:02.012007 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745390-mlfdw" event={"ID":"18f084fd-2c7b-4da3-b095-9db4cbf1e06b","Type":"ContainerDied","Data":"fb5c0bad7616a43b1ec00bb0ec53a70583a97d5198cae93aede836afbeaacf46"} Jul 22 12:30:03.160187 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:03.160156 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745390-mlfdw" Jul 22 12:30:03.199885 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:03.199859 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62" Jul 22 12:30:03.203182 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:03.203155 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb" Jul 22 12:30:04.022439 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:04.022400 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745390-mlfdw" Jul 22 12:30:04.022634 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:04.022399 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745390-mlfdw" event={"ID":"18f084fd-2c7b-4da3-b095-9db4cbf1e06b","Type":"ContainerDied","Data":"65e882afd2e118844acb262b5acd61269afb807b22b1462204c228285f8a9925"} Jul 22 12:30:04.022634 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:04.022556 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65e882afd2e118844acb262b5acd61269afb807b22b1462204c228285f8a9925" Jul 22 12:30:04.023740 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:04.023714 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62" Jul 22 12:30:04.023898 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:04.023750 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62" event={"ID":"a83064a4-e9ba-43b7-a3aa-5ed9fb01bd1f","Type":"ContainerDied","Data":"cdbecb325ca5c8930b3f12b6f2ffda225fb844005dc759a74c01e2b4c5666fd3"} Jul 22 12:30:04.023898 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:04.023799 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdbecb325ca5c8930b3f12b6f2ffda225fb844005dc759a74c01e2b4c5666fd3" Jul 22 12:30:04.025162 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:04.025134 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb" event={"ID":"20114a93-6b95-4619-bfb3-a386a6ffba57","Type":"ContainerDied","Data":"2b5ff735739f5486f2aa2b2cf25068bcfce3c27932cdd57213ff8ffa6c4d37a3"} Jul 22 12:30:04.025162 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:04.025152 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb" Jul 22 12:30:04.025301 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:04.025183 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b5ff735739f5486f2aa2b2cf25068bcfce3c27932cdd57213ff8ffa6c4d37a3" Jul 22 12:30:04.681467 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:04.681431 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:30:18.592524 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:18.592488 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:30:37.985911 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:37.985873 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:30:55.583658 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:30:55.583618 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:31:13.595753 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:13.595714 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:31:28.371046 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:28.370985 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:31:29.931035 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:29.930960 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:31:31.980968 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:31.980934 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:31:36.427624 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.427587 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl"] Jul 22 12:31:36.428152 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.428031 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20114a93-6b95-4619-bfb3-a386a6ffba57" containerName="cleanup" Jul 22 12:31:36.428152 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.428051 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="20114a93-6b95-4619-bfb3-a386a6ffba57" containerName="cleanup" Jul 22 12:31:36.428152 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.428064 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83064a4-e9ba-43b7-a3aa-5ed9fb01bd1f" containerName="cleanup" Jul 22 12:31:36.428152 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.428074 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83064a4-e9ba-43b7-a3aa-5ed9fb01bd1f" containerName="cleanup" Jul 22 12:31:36.428152 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.428093 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18f084fd-2c7b-4da3-b095-9db4cbf1e06b" containerName="cleanup" Jul 22 12:31:36.428152 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.428102 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="18f084fd-2c7b-4da3-b095-9db4cbf1e06b" containerName="cleanup" Jul 22 12:31:36.428368 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.428215 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="18f084fd-2c7b-4da3-b095-9db4cbf1e06b" containerName="cleanup" Jul 22 12:31:36.428368 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.428230 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="20114a93-6b95-4619-bfb3-a386a6ffba57" containerName="cleanup" Jul 22 12:31:36.428368 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.428242 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83064a4-e9ba-43b7-a3aa-5ed9fb01bd1f" containerName="cleanup" Jul 22 12:31:36.431166 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.431149 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.434616 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.434592 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0b54ae\"/\"sub-test-model-0b54ae-kserve-self-signed-certs\"" Jul 22 12:31:36.434746 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.434634 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0b54ae\"/\"kube-root-ca.crt\"" Jul 22 12:31:36.434746 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.434642 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0b54ae\"/\"openshift-service-ca.crt\"" Jul 22 12:31:36.434746 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.434618 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0b54ae\"/\"default-dockercfg-t29dw\"" Jul 22 12:31:36.444497 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.444470 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl"] Jul 22 12:31:36.456223 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.456186 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:31:36.468073 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.468045 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-tls-certs\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.468244 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.468097 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l64tw\" (UniqueName: \"kubernetes.io/projected/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-kube-api-access-l64tw\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.468244 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.468145 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-tmp-dir\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.468244 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.468184 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-dshm\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.468244 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.468200 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-model-cache\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.468244 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.468216 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-home\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.568878 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.568841 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-dshm\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.568878 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.568879 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-model-cache\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.569149 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.568906 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-home\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.569149 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.568937 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-tls-certs\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.569149 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.569006 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64tw\" (UniqueName: \"kubernetes.io/projected/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-kube-api-access-l64tw\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.569149 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.569033 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-tmp-dir\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.569369 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.569335 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-model-cache\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.569429 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.569405 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-home\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.569471 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.569444 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-tmp-dir\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.571282 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.571264 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-dshm\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.571420 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.571402 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-tls-certs\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.582670 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.582643 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l64tw\" (UniqueName: \"kubernetes.io/projected/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-kube-api-access-l64tw\") pod \"sub-test-model-0b54ae-kserve-578dbd584f-9gffl\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.740686 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.740601 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:36.877147 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:36.877099 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl"] Jul 22 12:31:36.880804 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:31:36.880774 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4254fd7_6ed5_4c5d_b2c3_c974e1d160c2.slice/crio-705099449eedf406f46a648ec07eefcb60de6537ebbd7248ce59e2d2d0891ebf WatchSource:0}: Error finding container 705099449eedf406f46a648ec07eefcb60de6537ebbd7248ce59e2d2d0891ebf: Status 404 returned error can't find the container with id 705099449eedf406f46a648ec07eefcb60de6537ebbd7248ce59e2d2d0891ebf Jul 22 12:31:37.452681 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:37.452641 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" event={"ID":"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2","Type":"ContainerStarted","Data":"33e2a34d715555a15568c53a3ae2a5cffa7856329fffd99e678d0db6aa729426"} Jul 22 12:31:37.452681 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:37.452683 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" event={"ID":"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2","Type":"ContainerStarted","Data":"705099449eedf406f46a648ec07eefcb60de6537ebbd7248ce59e2d2d0891ebf"} Jul 22 12:31:37.480806 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:37.480732 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" podStartSLOduration=1.480715822 podStartE2EDuration="1.480715822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:31:37.479300075 +0000 UTC m=+2520.228511682" watchObservedRunningTime="2026-07-22 12:31:37.480715822 +0000 UTC m=+2520.229927424" Jul 22 12:31:45.508071 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:45.508034 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:31:46.741355 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:46.741304 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:46.741776 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:46.741375 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:46.754219 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:46.754184 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:47.511181 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:47.511141 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:31:54.034983 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:54.034946 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:31:58.111009 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:31:58.110969 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:32:12.184298 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:12.184262 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:32:33.476214 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:33.476177 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:32:37.924872 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:37.924840 2555 scope.go:117] "RemoveContainer" containerID="34748cf845c1fcb3e76295b59c10ecf672bb02e9e829e645caede87738f5df99" Jul 22 12:32:39.332769 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:39.332736 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl"] Jul 22 12:32:39.357443 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:39.357408 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:32:40.826266 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:40.826231 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:32:49.293256 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:49.293215 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:32:52.276819 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:52.276775 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:32:54.353271 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.353230 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" podUID="d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2" containerName="main" containerID="cri-o://33e2a34d715555a15568c53a3ae2a5cffa7856329fffd99e678d0db6aa729426" gracePeriod=45 Jul 22 12:32:54.606707 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.606643 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:32:54.620255 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.620223 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l64tw\" (UniqueName: \"kubernetes.io/projected/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-kube-api-access-l64tw\") pod \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " Jul 22 12:32:54.620421 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.620303 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-home\") pod \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " Jul 22 12:32:54.620490 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.620472 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-tmp-dir\") pod \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " Jul 22 12:32:54.620555 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.620498 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-home" (OuterVolumeSpecName: "home") pod "d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2" (UID: "d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:32:54.620555 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.620510 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-model-cache\") pod \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " Jul 22 12:32:54.620660 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.620581 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-tls-certs\") pod \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " Jul 22 12:32:54.620660 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.620628 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-dshm\") pod \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\" (UID: \"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2\") " Jul 22 12:32:54.620660 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.620637 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2" (UID: "d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:32:54.620835 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.620752 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2" (UID: "d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:32:54.620885 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.620871 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-home\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:32:54.620921 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.620888 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-tmp-dir\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:32:54.620921 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.620898 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-model-cache\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:32:54.622518 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.622494 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-dshm" (OuterVolumeSpecName: "dshm") pod "d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2" (UID: "d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:32:54.622629 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.622541 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-kube-api-access-l64tw" (OuterVolumeSpecName: "kube-api-access-l64tw") pod "d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2" (UID: "d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2"). InnerVolumeSpecName "kube-api-access-l64tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:32:54.622629 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.622567 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2" (UID: "d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:32:54.721476 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.721439 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-tls-certs\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:32:54.721476 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.721468 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-dshm\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:32:54.721476 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.721478 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l64tw\" (UniqueName: \"kubernetes.io/projected/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2-kube-api-access-l64tw\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:32:54.805911 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.805871 2555 generic.go:356] "Generic (PLEG): container finished" podID="d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2" containerID="33e2a34d715555a15568c53a3ae2a5cffa7856329fffd99e678d0db6aa729426" exitCode=0 Jul 22 12:32:54.806087 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.805916 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" Jul 22 12:32:54.806087 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.805951 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" event={"ID":"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2","Type":"ContainerDied","Data":"33e2a34d715555a15568c53a3ae2a5cffa7856329fffd99e678d0db6aa729426"} Jul 22 12:32:54.806087 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.805995 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl" event={"ID":"d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2","Type":"ContainerDied","Data":"705099449eedf406f46a648ec07eefcb60de6537ebbd7248ce59e2d2d0891ebf"} Jul 22 12:32:54.806087 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.806013 2555 scope.go:117] "RemoveContainer" containerID="33e2a34d715555a15568c53a3ae2a5cffa7856329fffd99e678d0db6aa729426" Jul 22 12:32:54.815818 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.815797 2555 scope.go:117] "RemoveContainer" containerID="33e2a34d715555a15568c53a3ae2a5cffa7856329fffd99e678d0db6aa729426" Jul 22 12:32:54.816100 ip-10-0-130-190 kubenswrapper[2555]: E0722 12:32:54.816077 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e2a34d715555a15568c53a3ae2a5cffa7856329fffd99e678d0db6aa729426\": container with ID starting with 33e2a34d715555a15568c53a3ae2a5cffa7856329fffd99e678d0db6aa729426 not found: ID does not exist" containerID="33e2a34d715555a15568c53a3ae2a5cffa7856329fffd99e678d0db6aa729426" Jul 22 12:32:54.816280 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.816122 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e2a34d715555a15568c53a3ae2a5cffa7856329fffd99e678d0db6aa729426"} err="failed to get container status \"33e2a34d715555a15568c53a3ae2a5cffa7856329fffd99e678d0db6aa729426\": rpc error: code = NotFound desc = could not find container \"33e2a34d715555a15568c53a3ae2a5cffa7856329fffd99e678d0db6aa729426\": container with ID starting with 33e2a34d715555a15568c53a3ae2a5cffa7856329fffd99e678d0db6aa729426 not found: ID does not exist" Jul 22 12:32:54.832657 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.832625 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl"] Jul 22 12:32:54.844818 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:54.844789 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0b54ae/sub-test-model-0b54ae-kserve-578dbd584f-9gffl"] Jul 22 12:32:55.958957 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:32:55.958913 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2" path="/var/lib/kubelet/pods/d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2/volumes" Jul 22 12:33:16.337924 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:16.337839 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:33:19.989398 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:19.989359 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:33:26.953354 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:26.953317 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:33:27.053954 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:27.053914 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:33:32.685766 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:32.685726 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:33:35.383077 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:35.383046 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:33:37.943509 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:37.943474 2555 scope.go:117] "RemoveContainer" containerID="0a3b6898f1f89f13d9559d5dd58c7e8b7508083c08e4efeeb60f709cfe45b3b6" Jul 22 12:33:37.951648 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:37.951629 2555 scope.go:117] "RemoveContainer" containerID="4f27f2fde1e8fb4fdaa84e40d79572baabed80cc37b3200d3a086414a98a3772" Jul 22 12:33:48.700596 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:48.700555 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:33:48.798570 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:48.798535 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:33:52.118521 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:52.118488 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:33:56.931452 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:56.931415 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:33:57.084989 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:57.084959 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:33:58.955615 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:58.955574 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:33:59.912070 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:33:59.912036 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:00.776023 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:00.775988 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-139799750331312"] Jul 22 12:34:00.776432 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:00.776322 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2" containerName="main" Jul 22 12:34:00.776432 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:00.776334 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2" containerName="main" Jul 22 12:34:00.776432 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:00.776387 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4254fd7-6ed5-4c5d-b2c3-c974e1d160c2" containerName="main" Jul 22 12:34:00.779311 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:00.779295 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750331312" Jul 22 12:34:00.784933 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:00.784910 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h49wh\"" Jul 22 12:34:00.789899 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:00.789875 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-139799750331312"] Jul 22 12:34:00.861979 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:00.861942 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2bw\" (UniqueName: \"kubernetes.io/projected/e6abee69-4618-41b4-a7d4-2ee3677368cc-kube-api-access-cv2bw\") pod \"test-curl-7337-139799750331312\" (UID: \"e6abee69-4618-41b4-a7d4-2ee3677368cc\") " pod="opendatahub/test-curl-7337-139799750331312" Jul 22 12:34:00.963096 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:00.963065 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2bw\" (UniqueName: \"kubernetes.io/projected/e6abee69-4618-41b4-a7d4-2ee3677368cc-kube-api-access-cv2bw\") pod \"test-curl-7337-139799750331312\" (UID: \"e6abee69-4618-41b4-a7d4-2ee3677368cc\") " pod="opendatahub/test-curl-7337-139799750331312" Jul 22 12:34:00.976621 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:00.976584 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2bw\" (UniqueName: \"kubernetes.io/projected/e6abee69-4618-41b4-a7d4-2ee3677368cc-kube-api-access-cv2bw\") pod \"test-curl-7337-139799750331312\" (UID: \"e6abee69-4618-41b4-a7d4-2ee3677368cc\") " pod="opendatahub/test-curl-7337-139799750331312" Jul 22 12:34:01.087719 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:01.087687 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750331312" Jul 22 12:34:01.220603 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:01.220580 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-139799750331312"] Jul 22 12:34:01.222449 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:34:01.222419 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6abee69_4618_41b4_a7d4_2ee3677368cc.slice/crio-1125cb741864e331f6584c74122debddef4d6065e2b79145fab35982d299a127 WatchSource:0}: Error finding container 1125cb741864e331f6584c74122debddef4d6065e2b79145fab35982d299a127: Status 404 returned error can't find the container with id 1125cb741864e331f6584c74122debddef4d6065e2b79145fab35982d299a127 Jul 22 12:34:01.225609 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:01.224926 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 12:34:02.097150 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:02.097101 2555 generic.go:356] "Generic (PLEG): container finished" podID="e6abee69-4618-41b4-a7d4-2ee3677368cc" containerID="cd46272e08b8a515961e7303747675cb116e7197523c25deb818c5a78fa82d97" exitCode=0 Jul 22 12:34:02.097150 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:02.097137 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-139799750331312" event={"ID":"e6abee69-4618-41b4-a7d4-2ee3677368cc","Type":"ContainerDied","Data":"cd46272e08b8a515961e7303747675cb116e7197523c25deb818c5a78fa82d97"} Jul 22 12:34:02.097586 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:02.097177 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-139799750331312" event={"ID":"e6abee69-4618-41b4-a7d4-2ee3677368cc","Type":"ContainerStarted","Data":"1125cb741864e331f6584c74122debddef4d6065e2b79145fab35982d299a127"} Jul 22 12:34:03.236141 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.236087 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750331312" Jul 22 12:34:03.266520 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.266490 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7337-139799750331312_e6abee69-4618-41b4-a7d4-2ee3677368cc/test-curl-7337-139799750331312/0.log" Jul 22 12:34:03.281776 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.281743 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv2bw\" (UniqueName: \"kubernetes.io/projected/e6abee69-4618-41b4-a7d4-2ee3677368cc-kube-api-access-cv2bw\") pod \"e6abee69-4618-41b4-a7d4-2ee3677368cc\" (UID: \"e6abee69-4618-41b4-a7d4-2ee3677368cc\") " Jul 22 12:34:03.283981 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.283950 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6abee69-4618-41b4-a7d4-2ee3677368cc-kube-api-access-cv2bw" (OuterVolumeSpecName: "kube-api-access-cv2bw") pod "e6abee69-4618-41b4-a7d4-2ee3677368cc" (UID: "e6abee69-4618-41b4-a7d4-2ee3677368cc"). InnerVolumeSpecName "kube-api-access-cv2bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:34:03.323256 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.323224 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-139799750331312"] Jul 22 12:34:03.331121 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.331082 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-139799750331312"] Jul 22 12:34:03.383095 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.383012 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv2bw\" (UniqueName: \"kubernetes.io/projected/e6abee69-4618-41b4-a7d4-2ee3677368cc-kube-api-access-cv2bw\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:34:03.469574 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.469544 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-139799750107952"] Jul 22 12:34:03.469862 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.469850 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6abee69-4618-41b4-a7d4-2ee3677368cc" containerName="test-curl-7337-139799750331312" Jul 22 12:34:03.469909 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.469863 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6abee69-4618-41b4-a7d4-2ee3677368cc" containerName="test-curl-7337-139799750331312" Jul 22 12:34:03.469952 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.469924 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6abee69-4618-41b4-a7d4-2ee3677368cc" containerName="test-curl-7337-139799750331312" Jul 22 12:34:03.472799 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.472781 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750107952" Jul 22 12:34:03.480000 ip-10-0-130-190 kubenswrapper[2555]: E0722 12:34:03.479968 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-139799750107952\" is forbidden: User \"system:node:ip-10-0-130-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.ec2.internal' and this object" podUID="8a915b6f-5585-44c9-aad5-dcb31b0fe67d" pod="opendatahub/test-curl-7337-139799750107952" Jul 22 12:34:03.484317 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.484293 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn6z9\" (UniqueName: \"kubernetes.io/projected/8a915b6f-5585-44c9-aad5-dcb31b0fe67d-kube-api-access-qn6z9\") pod \"test-curl-7337-139799750107952\" (UID: \"8a915b6f-5585-44c9-aad5-dcb31b0fe67d\") " pod="opendatahub/test-curl-7337-139799750107952" Jul 22 12:34:03.500150 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.500123 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-139799750107952"] Jul 22 12:34:03.585442 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.585404 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn6z9\" (UniqueName: \"kubernetes.io/projected/8a915b6f-5585-44c9-aad5-dcb31b0fe67d-kube-api-access-qn6z9\") pod \"test-curl-7337-139799750107952\" (UID: \"8a915b6f-5585-44c9-aad5-dcb31b0fe67d\") " pod="opendatahub/test-curl-7337-139799750107952" Jul 22 12:34:03.609484 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.609447 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn6z9\" (UniqueName: \"kubernetes.io/projected/8a915b6f-5585-44c9-aad5-dcb31b0fe67d-kube-api-access-qn6z9\") pod \"test-curl-7337-139799750107952\" (UID: \"8a915b6f-5585-44c9-aad5-dcb31b0fe67d\") " pod="opendatahub/test-curl-7337-139799750107952" Jul 22 12:34:03.781500 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.781405 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750107952" Jul 22 12:34:03.923081 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.923047 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-139799750107952"] Jul 22 12:34:03.925542 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:34:03.925507 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a915b6f_5585_44c9_aad5_dcb31b0fe67d.slice/crio-ca4e4b8ef6fec9d5c64f1f270e2a7f9d0456220c4486afe266bb76acdcb23abd WatchSource:0}: Error finding container ca4e4b8ef6fec9d5c64f1f270e2a7f9d0456220c4486afe266bb76acdcb23abd: Status 404 returned error can't find the container with id ca4e4b8ef6fec9d5c64f1f270e2a7f9d0456220c4486afe266bb76acdcb23abd Jul 22 12:34:03.958606 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:03.958572 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6abee69-4618-41b4-a7d4-2ee3677368cc" path="/var/lib/kubelet/pods/e6abee69-4618-41b4-a7d4-2ee3677368cc/volumes" Jul 22 12:34:04.107409 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:04.107369 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-139799750107952" event={"ID":"8a915b6f-5585-44c9-aad5-dcb31b0fe67d","Type":"ContainerStarted","Data":"ca4e4b8ef6fec9d5c64f1f270e2a7f9d0456220c4486afe266bb76acdcb23abd"} Jul 22 12:34:04.108625 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:04.108602 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750331312" Jul 22 12:34:04.108625 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:04.108618 2555 scope.go:117] "RemoveContainer" containerID="cd46272e08b8a515961e7303747675cb116e7197523c25deb818c5a78fa82d97" Jul 22 12:34:05.114755 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:05.114710 2555 generic.go:356] "Generic (PLEG): container finished" podID="8a915b6f-5585-44c9-aad5-dcb31b0fe67d" containerID="7cbfe5f6861b8a8844478fcba792ac0f2b4ab92f7bdcee533e3323a1b4edee43" exitCode=0 Jul 22 12:34:05.115187 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:05.114792 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-139799750107952" event={"ID":"8a915b6f-5585-44c9-aad5-dcb31b0fe67d","Type":"ContainerDied","Data":"7cbfe5f6861b8a8844478fcba792ac0f2b4ab92f7bdcee533e3323a1b4edee43"} Jul 22 12:34:06.249073 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.249047 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750107952" Jul 22 12:34:06.305393 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.305354 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn6z9\" (UniqueName: \"kubernetes.io/projected/8a915b6f-5585-44c9-aad5-dcb31b0fe67d-kube-api-access-qn6z9\") pod \"8a915b6f-5585-44c9-aad5-dcb31b0fe67d\" (UID: \"8a915b6f-5585-44c9-aad5-dcb31b0fe67d\") " Jul 22 12:34:06.307465 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.307422 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a915b6f-5585-44c9-aad5-dcb31b0fe67d-kube-api-access-qn6z9" (OuterVolumeSpecName: "kube-api-access-qn6z9") pod "8a915b6f-5585-44c9-aad5-dcb31b0fe67d" (UID: "8a915b6f-5585-44c9-aad5-dcb31b0fe67d"). InnerVolumeSpecName "kube-api-access-qn6z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:34:06.309704 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.309682 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7337-139799750107952_8a915b6f-5585-44c9-aad5-dcb31b0fe67d/test-curl-7337-139799750107952/0.log" Jul 22 12:34:06.352282 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.352244 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-139799750107952"] Jul 22 12:34:06.357515 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.357490 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-139799750107952"] Jul 22 12:34:06.406451 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.406376 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn6z9\" (UniqueName: \"kubernetes.io/projected/8a915b6f-5585-44c9-aad5-dcb31b0fe67d-kube-api-access-qn6z9\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:34:06.540083 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.540053 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-139799750153248"] Jul 22 12:34:06.540435 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.540422 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a915b6f-5585-44c9-aad5-dcb31b0fe67d" containerName="test-curl-7337-139799750107952" Jul 22 12:34:06.540479 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.540437 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a915b6f-5585-44c9-aad5-dcb31b0fe67d" containerName="test-curl-7337-139799750107952" Jul 22 12:34:06.540513 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.540487 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a915b6f-5585-44c9-aad5-dcb31b0fe67d" containerName="test-curl-7337-139799750107952" Jul 22 12:34:06.543380 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.543362 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:06.556055 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.556031 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-139799750153248"] Jul 22 12:34:06.607606 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.607570 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tzgd\" (UniqueName: \"kubernetes.io/projected/de76f91e-87b4-4fc4-843f-fd4100bdeea1-kube-api-access-6tzgd\") pod \"test-curl-7337-139799750153248\" (UID: \"de76f91e-87b4-4fc4-843f-fd4100bdeea1\") " pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:06.708483 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.708384 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tzgd\" (UniqueName: \"kubernetes.io/projected/de76f91e-87b4-4fc4-843f-fd4100bdeea1-kube-api-access-6tzgd\") pod \"test-curl-7337-139799750153248\" (UID: \"de76f91e-87b4-4fc4-843f-fd4100bdeea1\") " pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:06.719849 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.719823 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tzgd\" (UniqueName: \"kubernetes.io/projected/de76f91e-87b4-4fc4-843f-fd4100bdeea1-kube-api-access-6tzgd\") pod \"test-curl-7337-139799750153248\" (UID: \"de76f91e-87b4-4fc4-843f-fd4100bdeea1\") " pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:06.851753 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.851716 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:06.986360 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:06.986334 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-139799750153248"] Jul 22 12:34:06.988327 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:34:06.988295 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde76f91e_87b4_4fc4_843f_fd4100bdeea1.slice/crio-b101fdc111c6be9d96bd83b32d984db4603e10082e6f3894defec5c450671144 WatchSource:0}: Error finding container b101fdc111c6be9d96bd83b32d984db4603e10082e6f3894defec5c450671144: Status 404 returned error can't find the container with id b101fdc111c6be9d96bd83b32d984db4603e10082e6f3894defec5c450671144 Jul 22 12:34:07.125838 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:07.125814 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750107952" Jul 22 12:34:07.126039 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:07.125821 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca4e4b8ef6fec9d5c64f1f270e2a7f9d0456220c4486afe266bb76acdcb23abd" Jul 22 12:34:07.126928 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:07.126902 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-139799750153248" event={"ID":"de76f91e-87b4-4fc4-843f-fd4100bdeea1","Type":"ContainerStarted","Data":"b101fdc111c6be9d96bd83b32d984db4603e10082e6f3894defec5c450671144"} Jul 22 12:34:07.135998 ip-10-0-130-190 kubenswrapper[2555]: E0722 12:34:07.135970 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-139799750107952\" is forbidden: User \"system:node:ip-10-0-130-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.ec2.internal' and this object" podUID="8a915b6f-5585-44c9-aad5-dcb31b0fe67d" pod="opendatahub/test-curl-7337-139799750107952" Jul 22 12:34:07.957417 ip-10-0-130-190 kubenswrapper[2555]: E0722 12:34:07.957382 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-139799750107952\" is forbidden: User \"system:node:ip-10-0-130-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.ec2.internal' and this object" podUID="8a915b6f-5585-44c9-aad5-dcb31b0fe67d" pod="opendatahub/test-curl-7337-139799750107952" Jul 22 12:34:07.959405 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:07.959383 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a915b6f-5585-44c9-aad5-dcb31b0fe67d" path="/var/lib/kubelet/pods/8a915b6f-5585-44c9-aad5-dcb31b0fe67d/volumes" Jul 22 12:34:08.133547 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:08.133509 2555 generic.go:356] "Generic (PLEG): container finished" podID="de76f91e-87b4-4fc4-843f-fd4100bdeea1" containerID="46d724caff7591f18222ebaa9bd52c4d34ab584a0f2d414f1bb56e3c2056c234" exitCode=0 Jul 22 12:34:08.133710 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:08.133594 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-139799750153248" event={"ID":"de76f91e-87b4-4fc4-843f-fd4100bdeea1","Type":"ContainerDied","Data":"46d724caff7591f18222ebaa9bd52c4d34ab584a0f2d414f1bb56e3c2056c234"} Jul 22 12:34:09.276464 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.276436 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:09.304010 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.303980 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7337-139799750153248_de76f91e-87b4-4fc4-843f-fd4100bdeea1/test-curl-7337-139799750153248/0.log" Jul 22 12:34:09.329333 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.329300 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tzgd\" (UniqueName: \"kubernetes.io/projected/de76f91e-87b4-4fc4-843f-fd4100bdeea1-kube-api-access-6tzgd\") pod \"de76f91e-87b4-4fc4-843f-fd4100bdeea1\" (UID: \"de76f91e-87b4-4fc4-843f-fd4100bdeea1\") " Jul 22 12:34:09.331937 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.331907 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de76f91e-87b4-4fc4-843f-fd4100bdeea1-kube-api-access-6tzgd" (OuterVolumeSpecName: "kube-api-access-6tzgd") pod "de76f91e-87b4-4fc4-843f-fd4100bdeea1" (UID: "de76f91e-87b4-4fc4-843f-fd4100bdeea1"). InnerVolumeSpecName "kube-api-access-6tzgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:34:09.332700 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.332678 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-139799750153248"] Jul 22 12:34:09.339020 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.338997 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-139799750153248"] Jul 22 12:34:09.430171 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.430084 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tzgd\" (UniqueName: \"kubernetes.io/projected/de76f91e-87b4-4fc4-843f-fd4100bdeea1-kube-api-access-6tzgd\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:34:09.459819 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.459788 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-139799750153248"] Jul 22 12:34:09.460100 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.460087 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de76f91e-87b4-4fc4-843f-fd4100bdeea1" containerName="test-curl-7337-139799750153248" Jul 22 12:34:09.460161 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.460101 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="de76f91e-87b4-4fc4-843f-fd4100bdeea1" containerName="test-curl-7337-139799750153248" Jul 22 12:34:09.460200 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.460185 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="de76f91e-87b4-4fc4-843f-fd4100bdeea1" containerName="test-curl-7337-139799750153248" Jul 22 12:34:09.462923 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.462904 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:09.469967 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.469946 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-139799750153248"] Jul 22 12:34:09.531236 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.531195 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d95bt\" (UniqueName: \"kubernetes.io/projected/af6aba49-0202-451f-b9ae-e006633b9cd0-kube-api-access-d95bt\") pod \"test-curl-7337-139799750153248\" (UID: \"af6aba49-0202-451f-b9ae-e006633b9cd0\") " pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:09.632019 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.631980 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d95bt\" (UniqueName: \"kubernetes.io/projected/af6aba49-0202-451f-b9ae-e006633b9cd0-kube-api-access-d95bt\") pod \"test-curl-7337-139799750153248\" (UID: \"af6aba49-0202-451f-b9ae-e006633b9cd0\") " pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:09.646305 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.646278 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d95bt\" (UniqueName: \"kubernetes.io/projected/af6aba49-0202-451f-b9ae-e006633b9cd0-kube-api-access-d95bt\") pod \"test-curl-7337-139799750153248\" (UID: \"af6aba49-0202-451f-b9ae-e006633b9cd0\") " pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:09.699904 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.699818 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:34:09.772630 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.772589 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:09.905482 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.905422 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-139799750153248"] Jul 22 12:34:09.907513 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:34:09.907490 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf6aba49_0202_451f_b9ae_e006633b9cd0.slice/crio-62c837adc31599d59d5eb7830078d4452be46269d9599c32280768f4d3a7b92f WatchSource:0}: Error finding container 62c837adc31599d59d5eb7830078d4452be46269d9599c32280768f4d3a7b92f: Status 404 returned error can't find the container with id 62c837adc31599d59d5eb7830078d4452be46269d9599c32280768f4d3a7b92f Jul 22 12:34:09.965910 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:09.965826 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de76f91e-87b4-4fc4-843f-fd4100bdeea1" path="/var/lib/kubelet/pods/de76f91e-87b4-4fc4-843f-fd4100bdeea1/volumes" Jul 22 12:34:10.145030 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:10.144997 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:10.145030 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:10.145021 2555 scope.go:117] "RemoveContainer" containerID="46d724caff7591f18222ebaa9bd52c4d34ab584a0f2d414f1bb56e3c2056c234" Jul 22 12:34:10.146305 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:10.146204 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-139799750153248" event={"ID":"af6aba49-0202-451f-b9ae-e006633b9cd0","Type":"ContainerStarted","Data":"62c837adc31599d59d5eb7830078d4452be46269d9599c32280768f4d3a7b92f"} Jul 22 12:34:11.152093 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:11.152058 2555 generic.go:356] "Generic (PLEG): container finished" podID="af6aba49-0202-451f-b9ae-e006633b9cd0" containerID="752d903a9c451003f0cc36cf6306f4a034a5be552f8a318fd9b2dca36efef454" exitCode=0 Jul 22 12:34:11.152567 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:11.152144 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-139799750153248" event={"ID":"af6aba49-0202-451f-b9ae-e006633b9cd0","Type":"ContainerDied","Data":"752d903a9c451003f0cc36cf6306f4a034a5be552f8a318fd9b2dca36efef454"} Jul 22 12:34:12.293875 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.293843 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:12.325246 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.325216 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7337-139799750153248_af6aba49-0202-451f-b9ae-e006633b9cd0/test-curl-7337-139799750153248/0.log" Jul 22 12:34:12.355923 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.355880 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-139799750153248"] Jul 22 12:34:12.356149 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.356129 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d95bt\" (UniqueName: \"kubernetes.io/projected/af6aba49-0202-451f-b9ae-e006633b9cd0-kube-api-access-d95bt\") pod \"af6aba49-0202-451f-b9ae-e006633b9cd0\" (UID: \"af6aba49-0202-451f-b9ae-e006633b9cd0\") " Jul 22 12:34:12.358196 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.358170 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6aba49-0202-451f-b9ae-e006633b9cd0-kube-api-access-d95bt" (OuterVolumeSpecName: "kube-api-access-d95bt") pod "af6aba49-0202-451f-b9ae-e006633b9cd0" (UID: "af6aba49-0202-451f-b9ae-e006633b9cd0"). InnerVolumeSpecName "kube-api-access-d95bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:34:12.361256 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.361231 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-139799750153248"] Jul 22 12:34:12.457505 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.457417 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d95bt\" (UniqueName: \"kubernetes.io/projected/af6aba49-0202-451f-b9ae-e006633b9cd0-kube-api-access-d95bt\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:34:12.499078 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.499045 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-139799750154256"] Jul 22 12:34:12.499385 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.499372 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af6aba49-0202-451f-b9ae-e006633b9cd0" containerName="test-curl-7337-139799750153248" Jul 22 12:34:12.499385 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.499386 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="af6aba49-0202-451f-b9ae-e006633b9cd0" containerName="test-curl-7337-139799750153248" Jul 22 12:34:12.499467 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.499436 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="af6aba49-0202-451f-b9ae-e006633b9cd0" containerName="test-curl-7337-139799750153248" Jul 22 12:34:12.502350 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.502334 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750154256" Jul 22 12:34:12.514840 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.514816 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-139799750154256"] Jul 22 12:34:12.558121 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.558079 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxpm9\" (UniqueName: \"kubernetes.io/projected/3d95b893-8ab3-4f72-8720-b39f3963d8c8-kube-api-access-bxpm9\") pod \"test-curl-7337-139799750154256\" (UID: \"3d95b893-8ab3-4f72-8720-b39f3963d8c8\") " pod="opendatahub/test-curl-7337-139799750154256" Jul 22 12:34:12.658552 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.658518 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxpm9\" (UniqueName: \"kubernetes.io/projected/3d95b893-8ab3-4f72-8720-b39f3963d8c8-kube-api-access-bxpm9\") pod \"test-curl-7337-139799750154256\" (UID: \"3d95b893-8ab3-4f72-8720-b39f3963d8c8\") " pod="opendatahub/test-curl-7337-139799750154256" Jul 22 12:34:12.670484 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.670450 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxpm9\" (UniqueName: \"kubernetes.io/projected/3d95b893-8ab3-4f72-8720-b39f3963d8c8-kube-api-access-bxpm9\") pod \"test-curl-7337-139799750154256\" (UID: \"3d95b893-8ab3-4f72-8720-b39f3963d8c8\") " pod="opendatahub/test-curl-7337-139799750154256" Jul 22 12:34:12.811265 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.811156 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750154256" Jul 22 12:34:12.949122 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:12.949086 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-139799750154256"] Jul 22 12:34:12.952414 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:34:12.952232 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d95b893_8ab3_4f72_8720_b39f3963d8c8.slice/crio-89ab56add85920d456fdbea6c9f16a59103c06a5a5589a457a9c218bc7d8cac1 WatchSource:0}: Error finding container 89ab56add85920d456fdbea6c9f16a59103c06a5a5589a457a9c218bc7d8cac1: Status 404 returned error can't find the container with id 89ab56add85920d456fdbea6c9f16a59103c06a5a5589a457a9c218bc7d8cac1 Jul 22 12:34:13.163975 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:13.163939 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-139799750154256" event={"ID":"3d95b893-8ab3-4f72-8720-b39f3963d8c8","Type":"ContainerStarted","Data":"89ab56add85920d456fdbea6c9f16a59103c06a5a5589a457a9c218bc7d8cac1"} Jul 22 12:34:13.165122 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:13.165088 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62c837adc31599d59d5eb7830078d4452be46269d9599c32280768f4d3a7b92f" Jul 22 12:34:13.165122 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:13.165117 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:13.175385 ip-10-0-130-190 kubenswrapper[2555]: E0722 12:34:13.175356 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-139799750153248\" is forbidden: User \"system:node:ip-10-0-130-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.ec2.internal' and this object" podUID="af6aba49-0202-451f-b9ae-e006633b9cd0" pod="opendatahub/test-curl-7337-139799750153248" Jul 22 12:34:13.964701 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:13.964662 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af6aba49-0202-451f-b9ae-e006633b9cd0" path="/var/lib/kubelet/pods/af6aba49-0202-451f-b9ae-e006633b9cd0/volumes" Jul 22 12:34:14.171011 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:14.170977 2555 generic.go:356] "Generic (PLEG): container finished" podID="3d95b893-8ab3-4f72-8720-b39f3963d8c8" containerID="9a679033168905777c2710fc95f7ac0f6a88280deaeb0ac3a017fde625bac299" exitCode=0 Jul 22 12:34:14.171238 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:14.171055 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-139799750154256" event={"ID":"3d95b893-8ab3-4f72-8720-b39f3963d8c8","Type":"ContainerDied","Data":"9a679033168905777c2710fc95f7ac0f6a88280deaeb0ac3a017fde625bac299"} Jul 22 12:34:15.310416 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:15.310392 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750154256" Jul 22 12:34:15.355539 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:15.355511 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7337-139799750154256_3d95b893-8ab3-4f72-8720-b39f3963d8c8/test-curl-7337-139799750154256/0.log" Jul 22 12:34:15.381825 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:15.381791 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxpm9\" (UniqueName: \"kubernetes.io/projected/3d95b893-8ab3-4f72-8720-b39f3963d8c8-kube-api-access-bxpm9\") pod \"3d95b893-8ab3-4f72-8720-b39f3963d8c8\" (UID: \"3d95b893-8ab3-4f72-8720-b39f3963d8c8\") " Jul 22 12:34:15.383834 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:15.383811 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d95b893-8ab3-4f72-8720-b39f3963d8c8-kube-api-access-bxpm9" (OuterVolumeSpecName: "kube-api-access-bxpm9") pod "3d95b893-8ab3-4f72-8720-b39f3963d8c8" (UID: "3d95b893-8ab3-4f72-8720-b39f3963d8c8"). InnerVolumeSpecName "kube-api-access-bxpm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:34:15.391845 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:15.391821 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-139799750154256"] Jul 22 12:34:15.399873 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:15.399849 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-139799750154256"] Jul 22 12:34:15.483090 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:15.483007 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxpm9\" (UniqueName: \"kubernetes.io/projected/3d95b893-8ab3-4f72-8720-b39f3963d8c8-kube-api-access-bxpm9\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:34:15.959058 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:15.959020 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d95b893-8ab3-4f72-8720-b39f3963d8c8" path="/var/lib/kubelet/pods/3d95b893-8ab3-4f72-8720-b39f3963d8c8/volumes" Jul 22 12:34:16.181941 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:16.181915 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-139799750154256" Jul 22 12:34:16.182146 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:16.181953 2555 scope.go:117] "RemoveContainer" containerID="9a679033168905777c2710fc95f7ac0f6a88280deaeb0ac3a017fde625bac299" Jul 22 12:34:19.019572 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.019538 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw"] Jul 22 12:34:19.019977 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.019847 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d95b893-8ab3-4f72-8720-b39f3963d8c8" containerName="test-curl-7337-139799750154256" Jul 22 12:34:19.019977 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.019859 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d95b893-8ab3-4f72-8720-b39f3963d8c8" containerName="test-curl-7337-139799750154256" Jul 22 12:34:19.019977 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.019911 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d95b893-8ab3-4f72-8720-b39f3963d8c8" containerName="test-curl-7337-139799750154256" Jul 22 12:34:19.023984 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.023962 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw" Jul 22 12:34:19.026780 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.026760 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kmhv\"" Jul 22 12:34:19.026947 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.026930 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 12:34:19.039296 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.039269 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw"] Jul 22 12:34:19.110381 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.110342 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce192cdb-3e7d-4513-8cf1-d6187690e187-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw\" (UID: \"ce192cdb-3e7d-4513-8cf1-d6187690e187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw" Jul 22 12:34:19.210795 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.210755 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce192cdb-3e7d-4513-8cf1-d6187690e187-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw\" (UID: \"ce192cdb-3e7d-4513-8cf1-d6187690e187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw" Jul 22 12:34:19.211412 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.211392 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce192cdb-3e7d-4513-8cf1-d6187690e187-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw\" (UID: \"ce192cdb-3e7d-4513-8cf1-d6187690e187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw" Jul 22 12:34:19.334055 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.334021 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw" Jul 22 12:34:19.488683 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.488651 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw"] Jul 22 12:34:19.491155 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:34:19.491129 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce192cdb_3e7d_4513_8cf1_d6187690e187.slice/crio-6c67fd6024f0368991b4479f432004e7f9ff5ff4cd3be1eace3a5301ad42fdbf WatchSource:0}: Error finding container 6c67fd6024f0368991b4479f432004e7f9ff5ff4cd3be1eace3a5301ad42fdbf: Status 404 returned error can't find the container with id 6c67fd6024f0368991b4479f432004e7f9ff5ff4cd3be1eace3a5301ad42fdbf Jul 22 12:34:19.808047 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.808010 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:19.907508 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:19.907420 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:20.208603 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:20.208506 2555 generic.go:356] "Generic (PLEG): container finished" podID="ce192cdb-3e7d-4513-8cf1-d6187690e187" containerID="96726f41eb0c0f5aebdf413106f86b32738f83b8fe34f349c249fd934d19c5e2" exitCode=0 Jul 22 12:34:20.208970 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:20.208592 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw" event={"ID":"ce192cdb-3e7d-4513-8cf1-d6187690e187","Type":"ContainerDied","Data":"96726f41eb0c0f5aebdf413106f86b32738f83b8fe34f349c249fd934d19c5e2"} Jul 22 12:34:20.208970 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:20.208645 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw" event={"ID":"ce192cdb-3e7d-4513-8cf1-d6187690e187","Type":"ContainerStarted","Data":"6c67fd6024f0368991b4479f432004e7f9ff5ff4cd3be1eace3a5301ad42fdbf"} Jul 22 12:34:21.341959 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:21.341933 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw" Jul 22 12:34:21.430698 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:21.430660 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce192cdb-3e7d-4513-8cf1-d6187690e187-maas-api-service-ca\") pod \"ce192cdb-3e7d-4513-8cf1-d6187690e187\" (UID: \"ce192cdb-3e7d-4513-8cf1-d6187690e187\") " Jul 22 12:34:21.431070 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:21.431042 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce192cdb-3e7d-4513-8cf1-d6187690e187-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ce192cdb-3e7d-4513-8cf1-d6187690e187" (UID: "ce192cdb-3e7d-4513-8cf1-d6187690e187"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:34:21.531854 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:21.531764 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce192cdb-3e7d-4513-8cf1-d6187690e187-maas-api-service-ca\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:34:22.218985 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:22.218905 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw" Jul 22 12:34:22.219164 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:22.218904 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw" event={"ID":"ce192cdb-3e7d-4513-8cf1-d6187690e187","Type":"ContainerDied","Data":"6c67fd6024f0368991b4479f432004e7f9ff5ff4cd3be1eace3a5301ad42fdbf"} Jul 22 12:34:22.219164 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:22.219041 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c67fd6024f0368991b4479f432004e7f9ff5ff4cd3be1eace3a5301ad42fdbf" Jul 22 12:34:29.507725 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:29.507621 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:29.514070 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:29.514045 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62"] Jul 22 12:34:29.517382 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:29.517360 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0b54ae-29745390-8ql62"] Jul 22 12:34:29.959033 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:29.959001 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a83064a4-e9ba-43b7-a3aa-5ed9fb01bd1f" path="/var/lib/kubelet/pods/a83064a4-e9ba-43b7-a3aa-5ed9fb01bd1f/volumes" Jul 22 12:34:32.779729 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:32.779697 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:34:34.104869 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:34.104837 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:34.113849 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:34.113821 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw"] Jul 22 12:34:34.117310 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:34.117287 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0b54ae-h8bhw"] Jul 22 12:34:34.505648 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:34.505566 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:35.958516 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:35.958481 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce192cdb-3e7d-4513-8cf1-d6187690e187" path="/var/lib/kubelet/pods/ce192cdb-3e7d-4513-8cf1-d6187690e187/volumes" Jul 22 12:34:37.977824 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:37.977797 2555 scope.go:117] "RemoveContainer" containerID="899a0dd2c27be05c02ac2a4488c6cf0a8a8c76ad05c95dd0916be96b9ee2447a" Jul 22 12:34:41.258837 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:41.258793 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:41.358531 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:41.358494 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:41.537591 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:41.537497 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:41.797035 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:41.796927 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:41.882256 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:41.882222 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc"] Jul 22 12:34:41.882608 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:41.882591 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce192cdb-3e7d-4513-8cf1-d6187690e187" containerName="revoke-keys" Jul 22 12:34:41.882690 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:41.882611 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce192cdb-3e7d-4513-8cf1-d6187690e187" containerName="revoke-keys" Jul 22 12:34:41.882750 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:41.882707 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce192cdb-3e7d-4513-8cf1-d6187690e187" containerName="revoke-keys" Jul 22 12:34:41.886143 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:41.886097 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc" Jul 22 12:34:41.888686 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:41.888666 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kmhv\"" Jul 22 12:34:41.888788 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:41.888712 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 12:34:41.894405 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:41.894381 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc"] Jul 22 12:34:41.999814 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:41.999780 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/368024b6-ddf3-4e5e-a4c5-82f261514e97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-55573a-pplkc\" (UID: \"368024b6-ddf3-4e5e-a4c5-82f261514e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc" Jul 22 12:34:42.101178 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:42.101139 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/368024b6-ddf3-4e5e-a4c5-82f261514e97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-55573a-pplkc\" (UID: \"368024b6-ddf3-4e5e-a4c5-82f261514e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc" Jul 22 12:34:42.101717 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:42.101696 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/368024b6-ddf3-4e5e-a4c5-82f261514e97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-55573a-pplkc\" (UID: \"368024b6-ddf3-4e5e-a4c5-82f261514e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc" Jul 22 12:34:42.196727 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:42.196694 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc" Jul 22 12:34:42.332945 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:42.332912 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc"] Jul 22 12:34:42.336127 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:34:42.336083 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368024b6_ddf3_4e5e_a4c5_82f261514e97.slice/crio-c372b757b44910d958db6c7e3cbb3e4912dc8d68386dccfd0fca66c1d8dff72f WatchSource:0}: Error finding container c372b757b44910d958db6c7e3cbb3e4912dc8d68386dccfd0fca66c1d8dff72f: Status 404 returned error can't find the container with id c372b757b44910d958db6c7e3cbb3e4912dc8d68386dccfd0fca66c1d8dff72f Jul 22 12:34:42.576044 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:42.576009 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:42.676034 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:42.675999 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:43.315808 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:43.315715 2555 generic.go:356] "Generic (PLEG): container finished" podID="368024b6-ddf3-4e5e-a4c5-82f261514e97" containerID="423b7d88e7c0c70f0ca5ec44c82dc44056742ca636f946f404166c8a1925c259" exitCode=0 Jul 22 12:34:43.315808 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:43.315756 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc" event={"ID":"368024b6-ddf3-4e5e-a4c5-82f261514e97","Type":"ContainerDied","Data":"423b7d88e7c0c70f0ca5ec44c82dc44056742ca636f946f404166c8a1925c259"} Jul 22 12:34:43.315808 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:43.315798 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc" event={"ID":"368024b6-ddf3-4e5e-a4c5-82f261514e97","Type":"ContainerStarted","Data":"c372b757b44910d958db6c7e3cbb3e4912dc8d68386dccfd0fca66c1d8dff72f"} Jul 22 12:34:44.458356 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:44.458330 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc" Jul 22 12:34:44.625972 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:44.625943 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/368024b6-ddf3-4e5e-a4c5-82f261514e97-maas-api-service-ca\") pod \"368024b6-ddf3-4e5e-a4c5-82f261514e97\" (UID: \"368024b6-ddf3-4e5e-a4c5-82f261514e97\") " Jul 22 12:34:44.626338 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:44.626313 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/368024b6-ddf3-4e5e-a4c5-82f261514e97-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "368024b6-ddf3-4e5e-a4c5-82f261514e97" (UID: "368024b6-ddf3-4e5e-a4c5-82f261514e97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:34:44.727377 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:44.727343 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/368024b6-ddf3-4e5e-a4c5-82f261514e97-maas-api-service-ca\") on node \"ip-10-0-130-190.ec2.internal\" DevicePath \"\"" Jul 22 12:34:45.326477 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:45.326430 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc" event={"ID":"368024b6-ddf3-4e5e-a4c5-82f261514e97","Type":"ContainerDied","Data":"c372b757b44910d958db6c7e3cbb3e4912dc8d68386dccfd0fca66c1d8dff72f"} Jul 22 12:34:45.326477 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:45.326467 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc" Jul 22 12:34:45.326707 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:45.326491 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c372b757b44910d958db6c7e3cbb3e4912dc8d68386dccfd0fca66c1d8dff72f" Jul 22 12:34:52.531314 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:52.531279 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb"] Jul 22 12:34:52.536747 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:52.536718 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-55573a-29745390-rm4pb"] Jul 22 12:34:52.551313 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:52.551286 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:53.958272 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:53.958232 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20114a93-6b95-4619-bfb3-a386a6ffba57" path="/var/lib/kubelet/pods/20114a93-6b95-4619-bfb3-a386a6ffba57/volumes" Jul 22 12:34:55.381032 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:55.380998 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mr49t"] Jul 22 12:34:57.006700 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:57.006665 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc"] Jul 22 12:34:57.010085 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:57.010059 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-55573a-pplkc"] Jul 22 12:34:57.010238 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:57.010190 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:57.438961 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:57.438929 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:34:57.962363 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:34:57.962331 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="368024b6-ddf3-4e5e-a4c5-82f261514e97" path="/var/lib/kubelet/pods/368024b6-ddf3-4e5e-a4c5-82f261514e97/volumes" Jul 22 12:35:05.382377 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:05.382332 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:35:05.809956 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:05.809862 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:35:11.364493 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:11.364462 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-756sh_5a4ed0bd-8ef3-4b59-8219-b82964e8fa13/manager/0.log" Jul 22 12:35:11.485240 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:11.485215 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-b9dk8_c291c1d6-703b-41c6-840d-cad9a4316686/manager/0.log" Jul 22 12:35:11.741657 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:11.741581 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-brfbg_1d911d66-0273-4aaa-b08b-264829357d6f/server/0.log" Jul 22 12:35:11.864655 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:11.864622 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-29bk6_db6bec41-3b3b-4cff-ae06-9091be5196fc/manager/0.log" Jul 22 12:35:12.113082 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:12.113057 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c4d6d56dd-dbzrv_6476de55-5def-4b98-a148-8bfc54acf4d8/copy-manifests/0.log" Jul 22 12:35:12.126785 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:12.126756 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c4d6d56dd-dbzrv_6476de55-5def-4b98-a148-8bfc54acf4d8/manager/0.log" Jul 22 12:35:12.991267 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:12.991239 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745390-mlfdw_18f084fd-2c7b-4da3-b095-9db4cbf1e06b/cleanup/0.log" Jul 22 12:35:14.642087 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:14.642059 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hr6xw_c870d06d-a4b3-452b-9de1-f3bd939abe36/manager/0.log" Jul 22 12:35:15.272515 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:15.272478 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-mr49t_3ef3f9e8-b94c-476f-a86a-12c28c9e7741/limitador/0.log" Jul 22 12:35:15.931212 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:15.931184 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2rjcx_989d2dd9-95f1-4166-bd35-21c6214d665e/discovery/0.log" Jul 22 12:35:16.170726 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:16.170701 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57559476c9-vrvbz_546779a6-4029-41ff-bedf-94bd239f0cbc/kube-auth-proxy/0.log" Jul 22 12:35:17.601790 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:17.601764 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-tdvr6_5177e6a9-f007-499f-8d67-aefed6851e13/main/0.log" Jul 22 12:35:23.491770 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:23.491736 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:35:23.594747 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:23.594711 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" limit=1024 Jul 22 12:35:27.594127 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:27.594084 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t4dxk_5465cf73-cdb9-46e8-ad04-70550bc615a8/global-pull-secret-syncer/0.log" Jul 22 12:35:27.673875 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:27.673844 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4s464_57933426-ecf3-46ae-87b9-9a9f8329462e/konnectivity-agent/0.log" Jul 22 12:35:27.817342 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:27.817309 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-190.ec2.internal_a0028e82bce45bbfeac23212d15624aa/haproxy/0.log" Jul 22 12:35:33.779823 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:33.779790 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hr6xw_c870d06d-a4b3-452b-9de1-f3bd939abe36/manager/0.log" Jul 22 12:35:34.660233 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:34.660096 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-mr49t_3ef3f9e8-b94c-476f-a86a-12c28c9e7741/limitador/0.log" Jul 22 12:35:36.526150 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:36.526122 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vtpbp_c32bd875-338b-4c02-93ac-b1b1bb5c96da/node-exporter/0.log" Jul 22 12:35:36.553358 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:36.553334 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vtpbp_c32bd875-338b-4c02-93ac-b1b1bb5c96da/kube-rbac-proxy/0.log" Jul 22 12:35:36.580540 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:36.580512 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vtpbp_c32bd875-338b-4c02-93ac-b1b1bb5c96da/init-textfile/0.log" Jul 22 12:35:37.025867 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.025830 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc"] Jul 22 12:35:37.026169 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.026156 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="368024b6-ddf3-4e5e-a4c5-82f261514e97" containerName="revoke-keys" Jul 22 12:35:37.026169 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.026171 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="368024b6-ddf3-4e5e-a4c5-82f261514e97" containerName="revoke-keys" Jul 22 12:35:37.026257 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.026235 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="368024b6-ddf3-4e5e-a4c5-82f261514e97" containerName="revoke-keys" Jul 22 12:35:37.029084 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.029067 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.031758 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.031736 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtjsz\"/\"openshift-service-ca.crt\"" Jul 22 12:35:37.031889 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.031737 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mtjsz\"/\"default-dockercfg-p49nt\"" Jul 22 12:35:37.031957 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.031736 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtjsz\"/\"kube-root-ca.crt\"" Jul 22 12:35:37.039069 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.039049 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc"] Jul 22 12:35:37.173456 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.173409 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33d34f19-e5f4-41dd-94a9-67c2a5f08045-proc\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.173456 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.173460 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5rc\" (UniqueName: \"kubernetes.io/projected/33d34f19-e5f4-41dd-94a9-67c2a5f08045-kube-api-access-zk5rc\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.173683 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.173537 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33d34f19-e5f4-41dd-94a9-67c2a5f08045-podres\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.173683 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.173586 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33d34f19-e5f4-41dd-94a9-67c2a5f08045-lib-modules\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.173683 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.173609 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33d34f19-e5f4-41dd-94a9-67c2a5f08045-sys\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.274918 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.274863 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33d34f19-e5f4-41dd-94a9-67c2a5f08045-lib-modules\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.274918 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.274925 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33d34f19-e5f4-41dd-94a9-67c2a5f08045-sys\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.275221 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.274976 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33d34f19-e5f4-41dd-94a9-67c2a5f08045-proc\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.275221 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.275001 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5rc\" (UniqueName: \"kubernetes.io/projected/33d34f19-e5f4-41dd-94a9-67c2a5f08045-kube-api-access-zk5rc\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.275221 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.275046 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33d34f19-e5f4-41dd-94a9-67c2a5f08045-podres\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.275221 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.275051 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33d34f19-e5f4-41dd-94a9-67c2a5f08045-sys\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.275221 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.275061 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33d34f19-e5f4-41dd-94a9-67c2a5f08045-proc\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.275221 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.275058 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33d34f19-e5f4-41dd-94a9-67c2a5f08045-lib-modules\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.275221 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.275176 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33d34f19-e5f4-41dd-94a9-67c2a5f08045-podres\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.284233 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.284189 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5rc\" (UniqueName: \"kubernetes.io/projected/33d34f19-e5f4-41dd-94a9-67c2a5f08045-kube-api-access-zk5rc\") pod \"perf-node-gather-daemonset-vgrvc\" (UID: \"33d34f19-e5f4-41dd-94a9-67c2a5f08045\") " pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.339428 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.339393 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.473218 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.473190 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc"] Jul 22 12:35:37.475895 ip-10-0-130-190 kubenswrapper[2555]: W0722 12:35:37.475868 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod33d34f19_e5f4_41dd_94a9_67c2a5f08045.slice/crio-942fff06db7ae57b228fa3e356ba0ffac2bbacec58c798388d420943d7bef538 WatchSource:0}: Error finding container 942fff06db7ae57b228fa3e356ba0ffac2bbacec58c798388d420943d7bef538: Status 404 returned error can't find the container with id 942fff06db7ae57b228fa3e356ba0ffac2bbacec58c798388d420943d7bef538 Jul 22 12:35:37.567261 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.567231 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" event={"ID":"33d34f19-e5f4-41dd-94a9-67c2a5f08045","Type":"ContainerStarted","Data":"6b8c19308b3deca42667b187cbfc3c3de5890d04c6e21c5a8819eb37e2d4dbb5"} Jul 22 12:35:37.567652 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.567280 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" event={"ID":"33d34f19-e5f4-41dd-94a9-67c2a5f08045","Type":"ContainerStarted","Data":"942fff06db7ae57b228fa3e356ba0ffac2bbacec58c798388d420943d7bef538"} Jul 22 12:35:37.567652 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.567347 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:37.587976 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:37.587929 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" podStartSLOduration=0.587912805 podStartE2EDuration="587.912805ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:35:37.586593751 +0000 UTC m=+2760.335805353" watchObservedRunningTime="2026-07-22 12:35:37.587912805 +0000 UTC m=+2760.337124406" Jul 22 12:35:41.435330 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:41.435301 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h425l_68931960-b129-4f9d-bc5a-a10e10c5757b/dns/0.log" Jul 22 12:35:41.460638 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:41.460608 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h425l_68931960-b129-4f9d-bc5a-a10e10c5757b/kube-rbac-proxy/0.log" Jul 22 12:35:41.653735 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:41.653709 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vclwc_41bf461d-75a8-4bed-8d22-bb7cdd9bf266/dns-node-resolver/0.log" Jul 22 12:35:42.262277 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:42.262245 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7fff7d8d76-k746k_a6c35d70-d0ce-43fb-acfd-37c85a4d2164/registry/0.log" Jul 22 12:35:42.381480 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:42.381452 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrmnr_20f96103-fe3f-4f0f-8f59-351a00064147/node-ca/0.log" Jul 22 12:35:43.569185 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:43.569151 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2rjcx_989d2dd9-95f1-4166-bd35-21c6214d665e/discovery/0.log" Jul 22 12:35:43.580954 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:43.580930 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mtjsz/perf-node-gather-daemonset-vgrvc" Jul 22 12:35:43.629319 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:43.629291 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57559476c9-vrvbz_546779a6-4029-41ff-bedf-94bd239f0cbc/kube-auth-proxy/0.log" Jul 22 12:35:44.575934 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:44.575903 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bfjgh_4b71e182-de9c-49c1-802b-312441501177/serve-healthcheck-canary/0.log" Jul 22 12:35:45.356530 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:45.356501 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pz6s9_ae05e4be-c308-457f-aaeb-c47ae8637746/kube-rbac-proxy/0.log" Jul 22 12:35:45.384523 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:45.384494 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pz6s9_ae05e4be-c308-457f-aaeb-c47ae8637746/exporter/0.log" Jul 22 12:35:45.412313 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:45.412285 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pz6s9_ae05e4be-c308-457f-aaeb-c47ae8637746/extractor/0.log" Jul 22 12:35:47.550629 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:47.550540 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2rjcx_989d2dd9-95f1-4166-bd35-21c6214d665e/discovery/0.log" Jul 22 12:35:47.582049 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:47.582023 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57559476c9-vrvbz_546779a6-4029-41ff-bedf-94bd239f0cbc/kube-auth-proxy/0.log" Jul 22 12:35:47.938055 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:47.938032 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-756sh_5a4ed0bd-8ef3-4b59-8219-b82964e8fa13/manager/0.log" Jul 22 12:35:48.146481 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:48.146449 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-b9dk8_c291c1d6-703b-41c6-840d-cad9a4316686/manager/0.log" Jul 22 12:35:48.426865 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:48.426840 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-brfbg_1d911d66-0273-4aaa-b08b-264829357d6f/server/0.log" Jul 22 12:35:48.574204 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:48.574179 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-29bk6_db6bec41-3b3b-4cff-ae06-9091be5196fc/manager/0.log" Jul 22 12:35:48.794554 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:48.794455 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c4d6d56dd-dbzrv_6476de55-5def-4b98-a148-8bfc54acf4d8/manager/0.log" Jul 22 12:35:48.839660 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:48.839633 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c4d6d56dd-dbzrv_6476de55-5def-4b98-a148-8bfc54acf4d8/copy-manifests/0.log" Jul 22 12:35:49.077693 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:49.077613 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2rjcx_989d2dd9-95f1-4166-bd35-21c6214d665e/discovery/0.log" Jul 22 12:35:49.109449 ip-10-0-130-190 kubenswrapper[2555]: I0722 12:35:49.109419 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57559476c9-vrvbz_546779a6-4029-41ff-bedf-94bd239f0cbc/kube-auth-proxy/0.log"